While I’ve worked on ops-only teams before, for the last three years I’ve been the ops guy on a team of devs. This changed two months ago when the formidable @fak3r joined us and now once again, ops is a team. When I was the sole ops guy, I become used to rolling hosts out with chef and so I’ve been blissfully in my own little world of ‘self-documenting’ ops code, however it turns out that while the code and builds with chef are self-documenting, chef itself isn’t, nor are the (many) legacy hosts that still exist in our cluster and some of the tools I’ve been using and taking for granted. Bringing on another team member always makes you re-evaluate how well documented things are. To complicate matters, at the same time as growing the team, we inherited a rather large second infrastructure to manage and consolidate into our existing infrastructure. On top of that, we’ve both just started working with the amazing RC team up at Harvard, so we’re both learning their infrastructure as well. Needless to say, documenting multiple new environments plus multiple legacy environments has proven challenging.

One of the awesome things about a new team member coming on board is always the new knowledge and perspectives they bring with them, new ways of looking at and solving problems and new tools and tricks to bring to the table. Before coming on board, @faker and I worked together on a few projects and had various ways of sharing info, from email and IM to Dropbox shared folders to google docs, google hangouts and wikis. By far the most useful way however, was when we would work on a problem and it would result in a blog post. This way, not only did we have the solution documented for the next time we needed quick access to it, but if anyone else had a similar problem, a short time googling might bring them to our solution. Whether it was something as simple as finding out that a university in the UK was the culprit in DoS'ing one of our sites, or rebooting XenServer machines on the command line or something as complex as solving global syncing problems for our huge datasets, the value of having other users input as well as being able to go back at any point and be reminded of the solution was obvious. It’s been way too long between posts here and I’m hoping to continue the tradition @fak3r started and get posts happening way more frequently up here. In this spirit, my next post will be on the awesomeness that is vagrant.