Dev is a Production Environment Too!

(Note: I originally made this post on my employer's tech blog.)

Prod is important–we have to keep it up!”

Yes, it’s of utmost importance. Is it the only important environment you support as a (Web) Operations team? How much attention do you devote to your Development and Staging/QA environments? What is the value you place in them?

The Benefits of Well Maintained Environments

Providing equal support to the quality of your ancillary environments can be quite advantageous. Even if you ensure a solid transitional process of supporting software upgrades from Dev through Staging to Production, you could still run into unnecessary cost-inducing issues.

Instead of having to debug oddities in software misconfigurations, engineers could have additional time to focus on ensuring the operationability of their apps. Wouldn’t it be great if you could increase the productivity of your users (developers), who then use that increase as a catalyst to give you better tools and APIs? Suddenly both groups exist in a symbiotic love-bubble. Amazing.

Living in an Agile World

It can sometimes be difficult for Ops folk to find their way in a scene where everyone around them is pushing for faster support response. We’re trying to keep production as stable as possible, keeping up with requests for multiple releases throughout the day. Engineers need a bunch of new software installed for the next version of their Whiz-Bang app, but how can we ever have time to get it all installed in multiple environments and test its security and stability?

Fortunately, there are a few possible solutions. The most obvious is using a configuration management tool such as Puppet (there are others, but Puppet is a personal favorite). Puppet is the perfect tool for an Agile Sysadmin living in an Agile World. It allows you to define specific resources on a node (host) in a declarative manner. Need to upgrade Ruby or the JVM on a box? Easy, just push a new manifest.

Another option is to require supporting software be bundled directly with the in-house application. This is pretty ingenious, because it offsets the burden of configuration to the software engineer. It also enforces a strict version dependency throughout the development life cycle. The affinity between the in-house app and its supporting software versions might easily get lost as the app gets pushed from Dev to QA and Staging to Prod.

There are a couple caveats to this method, however. For example, in relinquishing some control of configuration to Engineering the two groups must communicate more diligently to ensure the software is functioning properly. Possible security vulnerabilities of the supporting software must also be considered. Who is responsible for upgrading the software if it is checked into the source control repository instead of being installed directly on the host as part of the provisioning and/or release process?

Regardless of the method you choose, you will end up with a more reliable and easy to maintain set of environments. This will be a big win for everyone — Engineering, QA, Ops, the whole company.