This_IS_NEW_Post

my blog post

Agility: The Three Pillars

Not so long ago, the holy grail of IT Operations was keeping the system stable. Making sure no change accidentally occurs, and detecting if something goes wrong. Academic research explored root cause analysis, to find the source of a problem when it occurred. Convergence and Promise theory was built on the premise that there is a single state that we want the system to continuously converge to.

Then came the cloud, with cheap commodity servers that can instantly be used; with explosion of valuable APIs that can be composed quickly to bring new ideas to light; and  with companies such as Netflix, and Etsy that demonstrated how business value can be attained with innovation that is fueled by agility, namely,  the ability to react quickly to market needs.

Netflix and others, contributed to our understanding of “agility” by sharing the knowledge of what they did, and what they learned. The principles communicated, went well beyond “automate everything” (as advocated by the DevOps movement) and expanded to a new style for development, testing, and operations.

Following these well-publicized examples, today, it is well understood by startups that are “born on the cloud” that the how is as important as the what and this is because the what is not fixed, it continues to evolve and transform. The how is what allows continuous innovation. Therefore, how you implement your service, your DevOps process, your runtime analytics for insight matter! The most frustrating situation to be in for any app provider is to fail to make a necessary app change quickly enough, and thus lose the business to the competition. This can happen if the app or the DevOps processes are not designed from the beginning for maximum agility – a poor situation to be in.

Now what about enterprises? Many enterprises suffer from what is termed a “technology debt”; having to run and maintain a large code base that was written 20 or even 30 years ago, where the system bottlenecks, and optimization targets were totally different. Back in the old days, I/O was a bottleneck, and systems were written to optimize the transaction response time. Software was written as a monolith for maximum efficiency. Alas, monolith goes against the principles of agility – but back in the days agility was not a main concern, it was not the bottleneck!!

Today, Enterprises know they need to evolve and transform. Transformation will happen through identifying the part of the system that can and must be restructured according to the new principles, and integrated with the back end (that still requires the type of availability, consistency, security and performance that is achieved well with the system of records architecture).

Agility became a business imperative!

In many conversations I had with customers that are going through an enterprise digital transformation they are asking for three things (usually in the same first sentence): DevOps, Microservices, and Docker Containers.

I came to realize that DevOps, Microservices and Docker are the three main pillars to enable continuous innovation fueled by agility.  

DevOps was popularized in 2009 as a movement to address the gap between Developers and Operations in order to achieve agility with assurance. The important concepts include.

• Cultural Change – having Dev and Ops working together on a project since inception
• Everything as Code – The entire deployment process is automated, including the provisioning of the environment, build and deployment.

• Immutable infrastructure – Never make changes by hand, and as much as possible, never make changes “in place”. It is a good practice to stand up a new environment (“full stack deployments”) and gradually roll forward to it. 

Microservices is an architecture, delivery, and operations style based on the understanding that the release of your code is only the beginning of the journey (Read: “Release IT!” by Michael T Nygard). Software needs to be written in a way that makes it easy to maintain it and evolve it. The discussion around Microservices expands what the DevOps movement started to now also include the software architecture style for maximum agility with robustness. Important concepts include:

• Loose coupling – such that every component can evolve independently for maximum agility and robustness. The mechanisms used for loose coupling include:
  • Dynamic Binding – how services discover and bind
  • Dynamic Configuration – how to break implicit config dependencies between components
  • Data management – break data dependencies through data de-normalization and encapsulation
• Embrace failures – the realizations that failures can happen, and how we design our software, and test it to ensure that failures are discovered quickly, and the damage is minimized.

The last pillar for agility is Containers, and in particular Docker Containers.

Let me make one thing clear; Containers are not synonymous to Microservices; It is possible to deliver a service following the Microservices style using VMs (like Netflix did). So why is it that every company I talk with want both?

It is simple: Docker Containers are great enablers for agility for the following reasons:

• Fast start up time
• Single packaging across environments (dev, test prod) simplifies process and reduces opportunities for errors
• Dev centric eco system
• Effective use of infrastructure resources

It will be interesting to see how the space will continue to evolve with companies such as IBM announcing the Container Service with foundations services that enable agility with assurance optimized for Docker. For a full disclosure, I am not an innocent bystander. My IBM Research team contributed significantly to some of the differentiating features. More on that in future posts.