Agile Testing Days take-aways #3 - Infrastructure Testing with Jenkins, Puppet and Vagrant

Infrastructure-as-code (IaC) tools, such as Puppet and Chef, have fascinated me since I first became aware of them about three years ago, and I'm very excited that my organisation is now making use of them. Attending this session was thus an easy choice to make. Carlos Sanchez delivered a well balanced talk that was high-level enough for newbies to understand, but still provided enough meat to satisfy those familiar with the concepts. You can read the abstract and grab the slides from his blog.

I had two direct take-aways from his talk, one technical, the other cultural:

1. Carlos shared how RSpec-puppet can be used to assert that packages are set up correctly - think acceptance-test-driven-development for puppet.
2. There was a great quote and this picture at the very beginning, which gave the Printing Press as an example of a tool can lead to significant cultural change. The quote was from Patrick Debois: Tools can enable change in behaviour and eventually change culture. This is controversial point, because, in the case of BDD, for example, using tools too early can hinder or even prevent the right kind of culture change. Choosing the right tool at the right time is a difficult decision to make, but one we shouldn't be afraid of.

Following his talk, Carlos and I met up to have a deeper discussion on the following practices.

Environment specific variables and configuration servers

One of the benefits of IaC is that it helps overcome environmental configuration issues, by explicitly codifying each environment's configuration. A few years back, the concept of a configuration server was mooted, which also partly addressed this. The concept involves a central server and configuration repository, from which applications retrieve their configuration (via HTTP, for example), by telling the server where their running. We discussed how IaC solves this, and whether configuration servers are now redundant. Our conclusion was that environment specific variables (e.g. connection strings for QA, Pre-Prod, Live) should be separated from recipes into environment specific files. It's still worth considering the use of a configuration server, quite separate from Puppet/Chef. I think experimentation would be required to establish if this separation was beneficial or not. If any readers have experience of this, I'd love to hear it.

Should engineers develop on local, remote or cloud VM's?

IaC tools can lead to the increased use of Virtual Machines (VM's) throughout the development process, including an individual engineer's dev-environment. This raises the question of what's the best host for VM's used for this particular purpose. Is it better that they be hosted locally (on the engineer's own laptop), remotely on a VM Server, or remotely in the cloud (e.g. AWS). There are many trade-off's to make in deciding this, but our discussion concluded with the instinct that (beefy) local hosts were best.

What's best: immutable images or a clean base + recipes?

Martin Fowler has an excellent article that presents the concept of immutable images (a VM image that needs no configuration), and compares them to deploying binaries that need no further tinkering nor configuration. At first, this seems to go against the IaC concept of having base images that source-controlled recipes fully define the configuration of. When you evaluate this in practice though, a hybrid of the two practices gives the best of both worlds, as using an immutable image significantly speeds up chef execution and deployment. The immutable image should be created, though, by using a clean base image and a source-controlled set of recipes. The image and it's recipes should be updated every couple of weeks, depending on your release cycle.

Who chef's the chef / who puppet's the puppet?

Chef/Puppet recipes include definitions of required applications, where to get them from, and what version to get. So where do you define what version of chef to use (in order to keep it consistent and up-to-date)? Carlos suggested this could be achieved by a mix of running update scripts and using chef recipes themselves.