8th Light has accumulated a lot of servers, the cloud kind, over the years. Each one configured by the team that was building our internally commissioned software at the time, usually apprentices. And they did a good job setting up these servers, but some problems are brewing.
- The servers are not documented. If we were to lose a server in a crash, it would NOT be easy to recover.
- Each server is configured differently. No assumptions can be made when SSH-ing into a new server.
- There’s no catalog. Until recently, I couldn’t tell you how many servers we had or where they were located.
A bit scary, huh? It is for me. Now, this problem evolved over time. Each software project started out as an experiment. Some failed and those servers are gone, or off my radar. But the experiments that were successful, we slowly became dependent on those servers. And so here we are with this problem I just described.
Ironically, we’ve always known how to avoid the problem in the first place. For years we’ve been using tools like Puppet and Chef to automate server configuration for our clients. But the cobbler’s children don’t have shoes.
At 8th Light, we’ve undergone an exciting effort to consolidate all our servers using proper practices with modern tools. Although nothing that I’m about to describe is new or revolutionary, I feel it’s worth talking about because it is quite valuable, and perhaps underutilized.
In case you haven’t used Puppet or Chef, they are tools that allow you to write code that documents your server’s configuration with such detail that the code will execute and do the configuration for you. It’s brilliant! And if your servers are running on AWS or other cloud-based services, add Fog to the mix and you can also automate much of your cloud configuration too. That’s what we’ve done here at 8th Light. In fact, we took it one step further with Boucher so that we can control our entire cloud ecosystem via Rake tasks.
With this infrastructure in place, my fear is gone. We now benefit from a healthy list of advantages over our previous ecosystem.
0 to Deployed in 1 Command
Starting with nothing except an AWS account and some source code, I can execute 1 command that will
- create new server instances on EC2
- install all the required tools and packages
- download our software, and build it
- connect to the database(s)
- configure everything
- start it all
Yeah! Really! 1 command and an entire system or servers is running our software live on the web! I’m blown away when I think about it. To think that a decade ago my process was:
- order custom 1U units from Dell
- wait a week for delivery
- plug in servers at my desk
- manually install everything
- install servers in rack
- configure servers for installed environment
- start up services
We’ve come a long, long way! But I digress. Back to the advantages of using Boucher:
Complete Automation
For any reason small or big, I can effortlessly create a new instance of any server. And you might be surprised at how many reasons there are.
“Oh I wonder how site X would run with 2 Rails processes instead of 4?” Provision a new server and let’s find out!
“Joe is working on that server. I'll create a new one so I don't get in his way.”
Comprehensive Documentation
Every detail of every server is fully documented. Using Boucher, we know
- how many servers we have
- what services each server is running
- what dependencies each service has
- ….
Version History / Source Control
Server configuration gets first class treatment. It has its own source control repository and every change is tracked. This is a triumph. Take a note here: Huge success. It’s hard for me to overstate my satisfaction the significance of this.
Principles of Servers in Source Control
From working with Boucher and other projects where server configuration is automated with Chef or Puppet, I’ve collected a few principles that I’ll leave with you as a parting thought.
Hands Off!
No SSH-ing into the server to tweak configuration or install some package. The temptation is hard to resist, but you must remember that you’re trying to get your configuration code just right. Everything you do to the server manually is something that’s missing from your configuration recipe. It’s okay to SSH into a box to debug a problem or gather information, but your activity should remain READ-ONLY. Script EVERY action required to get the server in shape.
Idempotence
Your scripts/recipes must be idempotent. That is, you must be able to run and re-run your scripts without doing damage to the server or running services. When you think you’re done with a configuration script, run in on a brand new server. Make sure it works. Then run it again on the same server. Make sure the server still works.
The tools try to help you out here. Many of the built in tasks are already idempotent. And for custom tasks the tools provide only-if or not-if clauses where you can add code to check if your task needs to run.
Environments
You’ll need at least 2 environments in which you’ll provision servers. Production is the environment where stuff matters. You don’t want to try out configuration changes here. So you’ll need another environment, perhaps Staging or Test. These are environments where you can work on your scripts and try stuff out. Boucher provides support for environments and you should use it. Get your staging environment to mirror your production environment, or vice-versa. Then you’ll know that if your scripts run on staging without a hitch, they’ll almost certainly run fine on production.
