There has been a trend in applications and infrastructure away from optimization towards agility. Think SQL databases of yesterday – define your structures/tables, deploy them on a purpose built infrastructure, optimize your queries and tune your compute/network/storage – a lot of focus on optimization, but not very agile for the developer or the infrastructure administrator.

Today, the speed of innovation and ultimately the speed of doing business have necessitated a shift towards agility. NoSQL, the data lake, and cloud are all solutions driven by agility. But where does optimization fit in? Certainly some of it has been pushed into the application with sharded architectures and load balanced query scheduling. But that only gets so far, and as applications and data scale, optimization becomes a tougher challenge. It becomes apparent that agility and optimization are 2 sides of the same coin. When optimization falls behind, so does agility.

Maybe instead of “agility vs optimization”, it should be “optimized agility and agile optimization!” As discussed above, cloud-native applications and the cloud optimize agility. But what addresses agile optimization? Well, there’s software-defined <insert your resource here> and application aware infrastructure. The latter being the holy grail but has yet to be truly achieved in a general purpose deployment.

Today, optimization is still very manual, ad-hoc, and frequently requires re-implementation. That’s not agile optimization. Software-defined infrastructure is a start and is good at provisioning, but that’s a static definition.

What is really needed is something which, after initial provisioning, continues to adapt and enforce definitions as requirements change: agile optimization! And it needs to be general purpose across many applications, since the developer’s application toolkit continues to grow. If done right, the solution will have value to both developers and infrastructure administrators – 2 sides of another coin!

A lot of the pieces are here today like cloud native applications, containers, software defined infrastructure, big data, and most importantly, the cultural evolution of devops. Someone needs to put it all together. That’s a big vision and a big opportunity for innovators.