Monthly Archives: May 2012

Running IT like a Rock band

I’d like to take you on a little journey dear reader.

My background

I have two driving passions in my life. Music and computers. Since a kid I have tinkered with computers and noodled on musical instruments. For the past 10 or so years I have been working in IT, mostly as a sysadmin, mostly on busy websites. And I’ve been playing in bands, on and off. I’d like to draw some parallels between running an IT shop and playing in a rock band. Humour me?

Back in the day I went to University. I enrolled in a Science degree and studied Physics and Chemistry and all that jazz, and I took all the Computer Science classes. Mainly programming in C and networking. My first year assignments were done on real VT100 terminals connected to some wonderful old SGI and Sun hardware. Man, they were some good lookin’ servers! But the classes that really piqued my interest were those from the History and Philosophy of Science department, and a great CS subject called Professional Issues in Computing. I guess it started a long term interest in the culture that underpins our various pastimes and endeavours.

The analogy

So, in my day job, I’ve worked in Web 2.0 start-ups, small teams where everyone does everything through to bigger Enterprisy teams where people are specialised and siloed in departments. Although I studied to be a software developer, I have always been attracted to the more operational side of IT. I’ve hired and fired people and built teams. Have worked with some great people along the way. I’ve worked in some effective, collaborative environments and also in some dysfunctional ones. I’ve seen good environments go bad and even bad ones get good again. Outside of my day job I’ve played in a bunch of bands that have these characteristics too. I can’t help thinking there are some similarities.

I’ll use a typical rock band as a framework. 2 Guitars, bass, drums, vocals etc.

* The roles

The Drummer – Operations.
In the band, he’s the one who keeps the beat. He drives the music along and gives it energy. No other member of the band is under such constant pressure to perform. He rarely gets a break. If he’s missing it leaves a huge hole. You want him to be consistent and always available. Similarly the operations role (which may be shared in a small start-up, or a dedicated person, or team in a larger org) drives and maintains IT. Drummers and Ops don’t need a lot of glory and spotlight time, but they don’t like to be ignored either. They both like hardware.

Lead guitarist – Development.
The lead guitarist on the other hand does need to show off regularly. He’s practised till his fingers bled and needs to have his solos heard. Can be temperamental at times. Will definitely have strong opinions on how the song should go. The lead guitarist really adds colour and showmanship to the band but can have a tense relationship with the drummer. Sometimes gets ahead of the beat or even loses it. Probably ambitious and might be moonlighting with another band. Developers tend to move jobs more than operations. Interested in shiny things and new programming languages. Short attention spans, and don’t like repetition. However, they drive change and build features and need to be looked after.

Rhythm Guitarist – QA
Often overlooked. The rhythm guitarist is more comfortable out of the spotlight. Often highly accomplished and reliable though. They keep things together. They can back up the lead guitarist at any time, filling in gaps when the song’s getting out of hand. They listen to and speak the same language as the lead guitarist and the bass player. A small band/small IT shop may not have a dedicated rhythm/QA role. Instead, the responsibilities may be shared by others. Really good QA staff can catch bugs and performance regressions before they hit production. Big picture people who care for the end user.

Bass Player – DBA
The database administrator is a key bridge between operations and development. The bass player is the link between the drums and lead guitar. He has just as much commitment to the beat as the drummer, but has much to contribute to the melody and harmony too. Drums and guitars will definitely be listening to the bass player if they’re to play together well. The developer might find it convenient to SELECT * from bigtable but the DBA will insist on a WHERE clause for sane performance. Interesting characters if you get to know ’em.

Lead Singer – Product Owner
Now this is where things can really go wrong. Or right. The singer thinks they run the whole show. They have the personality and ego to be the main focus for the audience. They are the one who leads, who sets the style of the band, who gets the gigs, and writes the songs. They can be a temperamental prima donna, but audiences love them. Product owners can come into conflict with any in the organisation but watch out for their relationship between the lead guitarist/developer. Most important and they know it.

Keyboards – Security
Ok, how far will the analogy stretch… Not every rock band has a keyboard player. Nor does every IT shop have a security guy. The bigger ones of each do. Good to have one if you can.

Roadie – Network Engineer
Roadie sets things up before the show. Plugs in all the wires. Sets up lights, drums, smoke machines. Roadie makes sure the microphones and amplifiers are cabled. Network guy makes sure all the servers and switches are cabled. Probably knows a whole lot more than you think about how stuff works. Knows stuff about how stuff works that you don’t need to know. Arguably not needed so much when the band is playing (or the data centre is built), but if something goes wrong, it’s really really handy to have him around to fix it. Can fix a broken mic stand or router with duct tape in a crisis. Drives a fast car or motorcycle. Loves the hardware. Really loves the hardware.

* Songs, features.
What is it that a band or a typical IT organisation produce? Bands write songs. IT produces features on the website. Let’s say the lead singer is the creative force in the band and the Product Manager/Product Owner is the one accountable for driving the features. We can start to see immediately that the contributions of all members of the band or team are important. But if the creative vision is wrong, or the execution of the vision fails, it’s just not going to work well is it?

* What can go right
Ideally a harmonious working relationship exists between all members. If everyone’s pulling together, the band locks into the groove, the team morale is high, there’s nothing better. It’s a pleasure to play together or come to work. Amazing things can be accomplished. Hit records. Hit applications on the web. Fame and fortune.

A good leader realises that the contributions of all members of the team have worth. The singer may write most of the songs, but sometimes the drummer comes up with a hit tune too. Don’t disregard anyone on the team. Perhaps the QA guy has a better feel for which features will make money and which will annoy the crap out of the users.

A sign that the team is really working well together is that people start backing each other up, never undermining. In the band, members might cover each others parts if there’s a problem. In a work environment Ops might try being proactive about provisioning new development environments, or Dev might come over to chat with Ops about a new technology they’re thinking about implementing way in advance.

* What can go wrong
In the band, it’s often the prima donna lead singer, who is out of step with the rest of the band. They have creative differences and decide to go solo. Then they realise what they’ve lost and never capture the same vibe again. The singer/product owner needs to listen to what the rest of the band/team are saying. They need to rein in the ego off stage. There’s a tendency for these creative types to become too attached to their ideas. They might think a song is the best thing ever, or a new application feature is exactly what the users need. Sometimes they might be right. Sometimes they’ll be horribly wrong. The music industry has well established metrics to measure success or failure of a tune. Top 40 charts, downloads and the like. The IT industry have some catching up to do in this area. Smart players are using feature flipping and A/B testing to understand the impact of features.

In a band it’s pretty obvious when personnel are out of balance. The rock band formula has been around for more than 60 years. You don’t see (often) a band with 2 drummers. Or 10 guitarists. Yet I have sometimes seen a distinct lack of balance in IT. I’ve seen one team grow in numbers without adding the supporting head count for the other teams. For example, if Dev grows too fast they might become frustrated that Ops can’t keep up with their perfectly reasonable requests. If Ops proportionally outnumber Dev, they might also constrict the rate of innovation by adding too much process. Take a look at the balance of your wider team. Are there resource gaps, or even an oversupply? Too many Product Owners or persnickety Project Managers can be positively poisonous!

Some times in a band or a workplace there are just plain old personality clashes. Some people aren’t team players. They have their own career agendas and ambitions. Most people want to do good, but some, not many mind you, are just evil. Identify these people and work out if you can bear to spend a significant amount of your time with them and if not, leave before you become bitter and twisted yourself. Life’s to short to suffer assholism.

* Methodologies
Company culture and certain methodologies can vary widely depending on age and size.

An established band. They’ve been around for some years. Now they put on an occasional stadium show. They go into the recording studio and produce an album every 5 or so years. They’re worried about upsetting fans by changing too much.
Compare to an older Enterprise IT shop. They’ve built up processes and red tape. Have implemented ITIL. Play it slow and safe. They may still use a waterfall software development methodology and have armies of business analysts and project managers. They have legacy systems to maintain and are worried about changes or outages upsetting their users.

New band. They’re actively gigging around the traps. May be reliable and show up at the local pub on Saturday night. Or not. They’re producing songs quickly trying new sounds. They’re listening to their fans.
Like a fast young startup. Using any of lean, Agile, XP, scrum, Kanban.

* ITIL vs Devops
I guess it’s my amateur interest in organisational culture that has me excited by the devops movement, kicked off by maestro Patrick Debois a couple of years ago. I’ve seen first hand how big waterfall projects can fail to produce anything except used printer cartridges. And I’ve seen Agile fail too. Ultimately even more slowly and painfully. Agile and scrum methodologies are focussed on software development, and while great for delivering regularly incremental features, can miss the bigger picture. Agilistas need to ask themselves, are they continuously delivering the wrong things? Do they use the metrics Ops have or involve Ops in sprint planning? All the while they sang Kumbaya at stand up while the ship went down…

ITIL, Agile, Devops and Lean all try to give IT a clue to how to do things better. ITIL has so much potential, yet is so widely hated. I actually think ITIL is misunderstood and often misimplemented. The ideas are sound. It’s about delivering communication and change, safely. I’ve done the training. It’s not about filling in a form just to reboot a server. That’s a bad implementation of it. I think some managers see ITIL as about maintaining law and order in the wild west of IT land. About keeping the cowboys under control. It fails because smart people don’t like to be told what to do.

I think devops will succeed where others have failed because it is a holistic yet nebulous idea and defies definition.

But every devopstician has a devops definition anyway, so here’s mine. This week anyway.

Devops is putting the band back together. The C in CAMS is about culture, which I think is the most interesting bit of devops. It’s about an inclusive way of looking at IT and even the wider business (not just Dev and Ops) and getting teams working together again. It’s about software development and infrastructure and deployment, and feeding back data into product development so we end up building the right things. It’s a business solution not a technology problem (paraphrasing Damon Edwards). It’s about balance and harmony and having fun.


Filed under Uncategorized