As a sysadmin, how many times have you written a test before diving in and modifying a config file?
Never probably. That’s not typically how we do things.
But test driven development has been around for years and is used very effectively for building software. Why don’t we adopt the same rigor for our processes?
How about we try writing the equivalent of a unit test for a server configuration BEFORE starting to edit config files. Perhaps using Nagios (or our favorite testing / monitoring tool). Only when the screen turns from red to green, and our config is committed to our repo is our work is done.
Taking it a step further… Why not incorporate Nagios configs into our Continuous Integration environment. You do monitor your Test environment in the same way as Prod don’t you?!?
Process would look something like
1. Write (failing) test.
2. Make config change in test environment. (Ideally using configuration management with Chef, Puppet, Cfengine or the like).
3. Deploy change with software build to Test env.
4. Build succeeds, tests pass.
5. Refactor if necessary.
6. Proceed to Production. Profit.
This would require real DevOps cooperation. Peer review, pairing, communication!
Go on. Give it a try. Sysadmins need to earn the respect of developers by being just as good (if not better) at managing risk and change and also sharing what they do.