Monthly Archives: August 2012

Test driven systems administration

by David Lutz

The practice of test driven development has been around for some time and has proven to be a useful programming technique.

The general idea is
1. write a test case first (naturally it will fail)
2. write the minimal amount of code required to make the test pass (don’t over-engineer)
3. iterate and improve the code as required (without breaking the test)

As we are thinking about infrastructure as code it seems to make sense in theory to apply the same idea to systems administration.

In practise it’s not simple or straightforward. Assuming we’re using one of the popular frameworks for configuration management such as Chef or Puppet, there are a number of different ways of going about testing the code, and frankly, if you’re starting out it’s not at all clear which is the best way to do it.  If you’re using Chef for example to you look at minitest, cucumber-chef, test-kitchen, foodcritic? Something else? All of the above?  It’s a hot topic in the configuration management communities. Not a solved problem.

The learning curve for either Chef or Puppet is pretty steep. You have DSLs to learn, the Ruby programming language, unintuitive directory paths, servers, certs, convergence…

What if you don’t need something as complex as Chef/Puppet, don’t need a centralized server, but do want the ability to test stuff and write scripts that have “idempotent” properties?

First, let’s take a step back and see what we can do with plain old bash shell.

The example I’ll use is a typical task for a sysadmin. Install a package on a system. I’ve picked the text based web browser, links.

So let’s go.

Woah, hold your horses!

Let’s think about it from the perspective of a configuration management application. Actually we’ve made an assumption that the package is not already installed. As a human sysadmin wouldn’t you check to see if links is installed before blindly trying to install it again?

So what’s a nice way to check to see if an application is installed? We might just type the command, or use dpkg-query… how about the ‘which’ command? This’ll give us a nice return code that we can use programmatically.

so here’s our first go at test driven sysadmin in bash.

This script has a few properties of test driven systems administration.
It has a test. Is the package installed? I’m relying on the return code of which being 0 (true) if it is.
If the test passes the script does nothing. It’s safe to run multiple times.
If the test doesn’t pass, the script does something to make it pass. It installs the package.

It has some shortcomings. It assumes something about the environment. That it’s a debian or ubuntu based linux distribution. A better script would not make that assumption, but would work on other *nix environments. Portability is good. I typically work on OSX (BSD with homebrew) or Linux distributions with APT or RPM packages.

Ughhh, no one really writes scripts like this. It’s getting messy. The purpose of the script above is just to illustrate my point.

Enter Babushka. Test driven out of the box.

Babushka is a simple configuration management system. Sort of like Chef and Puppet in that the app can be used in a similar way to provision and configure a workstation, a development machine, or even a production server. But with a different focus. Babushka doesn’t have a server component. It’s a tool for automating the typical manual tasks a sysadmin would perform on a server.

The building blocks in Babushka are dependencies (aka deps). Which are either met or not. If a dependency is not met, (a failing test) then a block of code is executed to meet the dependency. Met and Meet!
Here’s how you’d approach the task of installing links with Babushka.

Create a subdirectory called babushka-deps

In this directory create a file. The name isn’t important but give it a .rb extension.
The format of the dep is something like this:

Now we can fill in our test (met?) and our action to take if it’s false (meet).

# babushka 'links is installed'
 links is installed {
 meet {
 Test failed. links is not installed, installing now...
 }
 } ✓ links is installed
# babushka 'links is installed'
 links is installed {
 } ✓ links is installed

Because installing a package is such a common task, there’s a platform independent way to do it. Like this. Change the name of the dep to dep.bin

.bin is a template. There are others.  .src .app

# babushka 'links.bin'
 links.bin {
 apt {
 package manager {
 'apt-get' runs from /usr/bin.
 } ✓ package manager
 apt source {
 } ✓ apt source
 apt source {
 } ✓ apt source
 } ✓ apt
 meet {
 Installing links via apt... done.
 }
 } ✓ links.bin

# babushka 'links.bin'
 links.bin {
 apt {
 package manager {
 'apt-get' runs from /usr/bin.
 } ✓ package manager
 apt source {
 } ✓ apt source
 apt source {
 } ✓ apt source
 } ✓ apt
 } ✓ links.bin

or to be platform independent

Or even

Now you don’t even need the do, end.

This will work on deb or rpm based Linux boxen or OSX. The important thing to remember is that even in its briefest form, the script above is following the test driven methodology. Test something and take action if the test fails.

The test can be something more complex than checking the return code from “which”. The idea is that you model it on the real commands you would type on the console. As a human sysadmin you might check for the existence of a file, or a process, or connect to a port, or otherwise check the response from a program.

Advertisements

Leave a comment

Filed under Uncategorized