This collection of blog posts is based on a talk I gave at Devoxx UK 2014, but the seeds for it were sown back in 2013. Since that time, trends in IT have continued to move on, and I’ve learned a great deal by putting much of this into practice on my own projects. Consequently, this collection is no simple rehash of the talk. No, it’s more akin to a revisit and re-evaluation. I hope it makes you question how you do things.

I described in the first post in this series how I think we’re witnessing a fundamental set of changes taking place in IT world today, both in the rise in complexity, and also in challenges to the “good ‘ole accepted wisdom” we’ve come to rely upon so much.  I ended by saying I’d start to look into some of the most interesting things to arise out of this maelstrom – informed by the lessons learned from the agronomy (yup, that’s farming) text ‘The One Straw Revolution‘. What follows is the first of these.

Part Two – The Bakery and NoOps

Let’s kick off by confronting our first truth about what we do:

Doing Things Causes Problems.

And it doesn’t matter whether we do these things manually or automatically. No matter what, we almost always seem to end up with problems.  But what I’d like to focus on here is not the traditional, technical specifics, but the overall Zen-like inevitability of it all – how depressing it is, and what we can do about it.

Stop Making Your Life Harder Than it Has to Be

In ‘The One Straw Revolution there is the admonishment to “do less” and to “leave things alone.”  If doing things inevitably causes problems, then perhaps to adopt this advice would perhaps be a great idea.  But how?

Well, one way is to further embrace the concept of immutability. We already know that immutability makes many things easier in the tech world:

  • Maintenance – in the good ‘ole days, Sun Microsystems had a dumb terminal called the SunRay. It had almost no moving parts, and almost nothing to update, and was described by some as a keyboard, mouse and monitor extension device – everything ran back on the server.  This meant it was cheap, and more importantly, that it failed very infrequently
  • Multi-Threading – we all know that multithreading is far easier if things are immutable, and it’s one reason why the Functional programming style has been gaining ground of late. If things can’t change, then another thread can’t change them underneath you – and that means less hairy bugs to got to the bottom of
  • Scaling – Scaling horizontally with many machines which are all of the same spec is a lot easier than scaling vertically by adding more grunt to your existing boxes. It’s just more predictable
  • Caching – aka “one of the two (three?) hardest things about software engineering.” Immutability is known to make cache invalidation far easier
  • Development – hell, we’ve even adopted immutability in development. Remember that Scrum insists we allow “no changes within a sprint.”

So with all this in mind, some clever folks out there on the internet got to thinking:

What if you had immutable deployables?

And taking it even further:

What if you deployed them to immutable environments?

These two tightly-coupled concepts form the basis of the rest of this post’s topic.

NoOps?  None? None at all?

What follows is taken from a long blog post by Adrian Cockroft entitled ‘Ops, DevOps and PaaS (NoOps) at Netflix‘.  It caused quite a stir in some communities, but we’re going to ignore that and look at some of the ideas it articulates.

Adrian writes;

“Netflix is a developer oriented culture.”

before continuing;

“We decided to leverage developer oriented tools such as Perforce for version control, Ivy for dependencies, Jenkins to automate the build process, Artifactory as the binary repository and to construct a “Bakery” that produces complete AMIs that contain all the code for a service.”

“Several hundred development engineers use these tools to build code, run it in a test account in AWS, then deploy it to production themselves.”

Lets bring these points out nice and clearly:

  1. They brought developer tools and processes as much to the centre of everything they did as possible
  2. Engineers then use these tools they already know well, and some more tools they built on top of them, to build and package…
  3. … and then deploy to PROD themselves.

Wait a minute. Don’t many of us work in “developer oriented cultures” too? Could we do this? Could we become just a bit like Netflix?  The answer is yes – because luckily for us, Netflix are very open about what they do, and even better, they share the code for much of it. Let’s take a quick look at the core elements.

Baking Images in the Bakery

Screen Shot 2015-01-09 at 11.31.10

[Image from]

First up is the “baking” that Adrian alluded to.  See the immutable base AMI images – and also notice how simple they are (Linux, Apache, Java and Tomcat – that’s it.)  And then observe them being baked into the immutable “app bundle” in the Bakery. And then notice the taking of a snapshot ready for deployment.

It’s redundant to go into this more when you can watch Carl doing the whole presentation over on InfoQ (where you can also download the slides).  Trust me, it’s worth the effort. I’ll wait here while you do it…

Developer-Self-Service with Asgard

Done? Good. Lets get back to Adrian again as he elaborates on the developer self-service deployments, introducing Asgard, a Groovy / Grails app built by Clay McCoy and friends:

“They [the Netflix Developers] use a web based portal to deploy hundreds of new instances running their new code.”

This GIF (from the Asgard announcement on the Netflix blog) shows this in action.

Adrian continues.

“Pushes to the cloud are as frequent as each team of developers needs them to be”

Before concluding:

“…running their new code alongside the old code, they put one “canary” instance into traffic, if it looks good the developer flips all the traffic to the new code.

“If there are any problems they flip the traffic back to the previous version (in seconds) and if it’s all running fine, some time later the old instances are automatically removed.”

You can see this in action here via another nice animated GIF from the Asgard announcement.

Again I’ve unfortunately not got the time or space to go into this in detail, but you can find out loads more by Googling “Asgard”, following them on twitter (@asgardoss) and attending their monthly Google hangouts

This is Great, But What’s the Point?

The point here is that, by building immutable deployables, using standard developer tools and tool chains, and by engineering them to be simply and automatically deployed, and undeployed via a tool like Asgard, Netflix have made their developers lives far, far easier.

In fact, they’ve taken advantage of the concept of immutability to such an extent that they also get traditionally “hard” things like continuous delivery, elastic scaling, and reduced their dependence on traditional, separate-team DevOps almost for free.  That is to say, by explicitly designing for this way of packaging, deploying and undeploying they’ve partially unlocked many other goals many other folks only dream of achieving.

Redux-update: What’s Happened Since Devoxx UK 2014

After the conference, cloud services (PAAS) continued their dominance, and perhaps more importantly, the near-ubiquity of the Microservices architecture style (seen in the arrival of Spring Boot/Cloud and the further increase in popularity of Ratpack, Dropwizard, Play etc.) combined with Docker (and its associated ecosystem) really took hold of the tech industries imagination.

The former facilitated moves towards the benefits of immutable deployables (for example as “Fat Jars”) and the latter towards immutable environments which can be set up and torn down with ease.  I’ve no doubt that very few people have made the final jump to developer-driven-deployment, but to move to this involves something far more complicated than adoption of a new technology or framework, this requires cultural change.  We’ll have to wait a little longer for that.

So What’s Coming Next?

That’s it for this iinstalment But up next is a very closely related topic: Antifragility and the Simian Army, where we’ll question more accepted wisdom, and see what else we can learn.


The 5 “Whys” Redux: Part Two – The Bakery and NoOps

About The Author
- A highly enthusiastic, self-starting and responsible software engineer/architect; Andrew specialises in Java / Java EE technologies, agile delivery, build tools and automation, and domain driven design. Experienced across the software development lifecycle and in many sectors including government, banking, and eCommerce, what motivates him is the production of a large-scale software solution, fulfilling complex client requirements. He understands that people, tooling, architecture and process all have key roles to play in achieving this. He greatly enjoys working in all these areas. Andrew has a passion for open source software and its communities; and they are central to his continuing professional and personal development and he currently runs the Capgemini UK OSS community. However, he has been interested in and involved with OSS to a greater or lesser extent since his career began; as a user, contributor, expert group member, or paid advocate. Finally, Andrew enjoys sharing his experience as much as possible. He is currently engaged in leading the creation of a Software Craftsmanship Course to help others adopt XP programming practices, but this sharing is also seen in his formal consulting engagements, and informally through mentoring undergraduates, blog posts, conferences, on forums, and open sourcing his code.

You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <s> <strike> <strong>