Monthly Archives: June 2011

Velocity 2011 day 1

Arrived nice and early at the Santa Clara convention center and missed the big queue for registration. (That’s a #protip folks!)

There was a real buzz at the first day of Velocity.  Lots of people who knew each other saying hi.  My fellow Aussies Mark Barger and Daniel Hall went to the Performance Tools Session with Steve Souders and friends.  Mark Jennings who was delayed by the Chilean volcano was somewhere in the air still.  Bitterly disappointed no doubt.

I attended the Netflix session with Adrian Cockcroft.  He spoke about Netflix’s scaling problems and solutions.  They are heavy users of Amazon’s infrastructure.  I really enjoyed this session.  I was expecting another cloud hypefest, but Adrian gave us many well reasoned insights into why they chose to use “the cloud”.  Why it made sense from a technical and financial perspective.

Netflix have some pretty impressive geek bragging rights.  I was excited (I admit it) by talk of thousands of servers and terabit data usage and playing CDN providers off against one another.

Adrian incorporated a 5 minute Ignite talk into his session on anti-architecture patterns.  This was awesome.  Really got me thinking.  Architectures so often go wrong, and software “Architects” so often have no idea what they’re doing.  If you clearly define what you don’t want  and establish the constraints up front (like it must be finished next week) and enforce those constraints, you may be more likely to end up with what you really want.

And not a DVD re-winder!

Session 2 I attended was the Opscode Chef workshop.  Delivered by Opscode.  All of them I think!

I love these guys and the Chef software, and use it already.

I’m not sure if it worked so well as a workshop though. Although the team’s delivery of the material was good and interesting and the tag team delivery was ok, I suspect someone who wasn’t familiar with Chef would have been a bit lost.  The pace was pretty fast.

It’s not that the demonstration wasn’t impressive (spin up 5 servers with monitoring, web/app and database) in about 5 minutes, completely configured.  It was.  I just feel they didn’t communicate just how awesome the tool is very well.

My takeaway was people who aren’t on 0.10 really should upgrade and use Environments.  These look cool.

After lunch was John Allspaw’s Session on Post Mortems and Human Error.

John is a great speaker and always has some awesome insights.

What I really like when I hear John speak is that he clearly articulates what it is to be in Web Operations and to be an Engineer.

That it is a serious and important profession and to be effective in the role you need to have experience in a range of engineering disciplines, and a good understanding of psychology too.

The feeling I get is that if you’re working at Etsy, the goals of those in Development and Operations are so closely aligned, that there is no Development and Operations.  There is only DevOps, that is, some Engineers that spend more time cutting code, and other Engineers who spend more time making graphs.

He described the knowledge gained from dealing with crises and outages as scar tissue, and that those with this experience are a huge asset to a company operating at scale.

I liked the philosophical and other research references sprinkled through his talk.  Clearly I have lots of homework to do.

The final session of day 1 I attended was from the Neustar guys.  Simon Nicoud, Mark Watson and Ian White.   

This is more like what I was expecting from a workshop.  CDs were handed out with some java and python code to drive Firefox via Selenium.

The guys walked through setting up a simple framework for automated web performance testing.  There are lots of tools and services around to do this kind of stuff, but I really liked the approach using simple tools to build something useful from scratch.


Wow.  Lots of good stuff here.

The “Experiment” was Ignite Karaoke.  Or more like Improv Stand-up.  Participants had to say something about some random slides. Time will tell if it was a successful experiment or not…  What do you think?


Leave a comment

Filed under Uncategorized

Test Driven Configuration


As a sysadmin, how many times have you written a test before diving in and modifying a config file?

Never probably. That’s not typically how we do things.

But test driven development has been around for years and is used very effectively for building software. Why don’t we adopt the same rigor for our processes?

How about we try writing the equivalent of a unit test for a server configuration BEFORE starting to edit config files. Perhaps using Nagios (or our favorite testing / monitoring tool). Only when the screen turns from red to green, and our config is committed to our repo is our work is done.

Taking it a step further…  Why not incorporate Nagios configs into our Continuous Integration environment.  You do monitor your Test environment in the same way as Prod don’t you?!?

Process would look something like
1. Write (failing) test.
2. Make config change in test environment. (Ideally using configuration management with Chef, Puppet, Cfengine or the like).
3. Deploy change with software build to Test env.
4. Build succeeds, tests pass.
5. Refactor if necessary.
6. Proceed to Production.  Profit.

This would require real DevOps cooperation.  Peer review, pairing, communication!

Go on. Give it a try. Sysadmins need to earn the respect of developers by being just as good (if not better) at managing risk and change and also sharing what they do.

Leave a comment

Filed under Uncategorized

DevOps friendly Organizational Structures

Playing Devil’s Advocate on the topic of “DevOps is not a job title“.

Here is a (simplified) traditional organizational structure.

Where red boxes represent Developers (or teams of developers) and blue ones represent Operations staff.

Let’s suppose the heads of Dev and Ops get on well and share common goals.  DevOps in this organization can work and probably is already.

But more often than not the heads of Dev and Ops don’t see exactly eye to eye.  If they don’t share common goals (feature delivery vs stability), if they are competing with each other for funds, attention, praise, the CIO’s job… DevOps may not be working so well.

Is this a familiar scenario?

If DevOps manager was a job title, with Developers and Operations staff reporting directly to one person, an org structure might instead look like this.

Suddenly Developers and Operations staff have common goals.  They share a manager.

Head count is the same.  DevOps managers are peers with programmers and sysadmins reporting to them.

The structure presented is a radical departure from traditional thinking but enables the benefit of a small start-up IT culture in the context of a larger company.

1 Comment

Filed under Uncategorized