Deploying your website manually is often a waste of time and energy. Here's how we used automation to forge a better path at Reflect.
I have a confession: I actually kind of enjoy manually deploying and managing
websites. I grab a cup of coffee, run
make publish or some similar command,
gaze into the CLI output, wait for static assets to be generated and
to a server thousands of miles away, and refresh the browser until I see the
new changes go live—one of life’s subtle pleasures.
But let’s face it, there’s usually no good reason to do things this way now that continuous deployment and similar automation patterns have become standard practice in our industry. We decided a while back here at Reflect to use the CI tool that we already use for everything else, CircleCI, to automate our website deployment flow, and the benefits were immediately clear.
Before we get into website deployment, a few words on how we build our site for context.
We use Jekyll as both our static site generator and as
the manager of our static asset pipeline.
Basically we feed a bunch of
fully built site in a
_site directory. We’ll say a lot more about this in a
Outside of this we we also use: Gulp.js for things like managing BrowserSync for local development; Rake for running our deployment scripts, which are heavily parameterized and benefit greatly from Ruby syntax (rather than Bash); and of course good old Make, which is sort of the main entry point into the project (our CircleCI jobs only ever run Make commands).
So where does our site run and how do we get it there? Here’s a quick overview of our setup:
Initially, our nginx configs simply lived on our droplets. If we
needed to update those configs we SSHed into the boxes and made ad hoc changes.
But the downsides of this are pretty clear: it’s easy to update one box and
forget to update the other, it’s annoying to have to jump onto another box just
to know what the config even is, and so on.
So we decided to store our nginx configs in the repo alongside the site itself, which means that they’re now subject to version control and easily viewable. So how do we make sure that nginx actually uses those configs? We’ll say more about that below.
All our code lives on GitHub, which we may know a thing or two about. We use CircleCI for everything CI related at Reflect. We have numerous repos hooked up to it, we get a steady flow of info from it on a dedicated Slack channel, and we’ve been very happy with it thus far.
The CircleCI configuration for our
website lives in a
circle.yml file (as per the convention) and is currently
machine: timezone: America/Los_Angeles node: version: 7.4.0 ruby: version: 2.3.1 dependencies: override: - make setup deployment: production: branch: master commands: - make release staging: branch: development commands: - make release_staging
This config specifies that:
America/Los_Angelestimezone, because that’s the one we all live in (we’re lobbying behind the scenes for an
America/Portlandtimezone…we’ll let you know how that goes)
Prior to the build, an additional
make setup command will be run. That
command looks like this in our
gem install bundler bundle install npm install
Although there are always exceptions and edge cases, for the most part all projects at Reflect follow a workflow in which:
masterbranch is considered the most up to date
developmentbranch rather than to
developmentis merged into
masteronly as part of a pull request, which enables us to review the collected result of many pull requests before we update
The website follows this flow as well. We’ve found this flow to be very
friendly to our continuous deployment patterns, as we can confidently push bold
development with the assurance that they’ll be vetted one more
time before being pushed to
master, which can mean anything from updating our
libraries in npm to updating Debian packages to, in
this case, pushing the website live.
Once the CI environment is all set up, what happens next depends on the branch to which we’re merging:
master, the production site is updated. That means that all of the following happens in CircleCI:
make releasecommand is run, which
rsyncs the contents of the
_sitefolder to both Digital Ocean droplets,
scps our nginx configuration files, and uses SSH to restart nginx and perform some basic housecleaning (like
chowning certain directories).
development, the staging version of the site is updated (this includes all of the steps for #1).
By default, CircleCI runs
make test whenever you push to a branch that
doesn’t have any behaviors associated with it. In our case, that means that
make test is run whenever we push to branches outside of
make test returns 0 the build passes; otherwise, the
At the moment, our
make test command does one thing:
make build, which
jekyll build. This process exits with a 0 (success) or
something else (failure). If we push anything to any branch and Jekyll can’t
build it then CircleCI will let us know via Slack immediately. In the future
we’d like to “test” our site more comprehensively by adding things like a
linkchecker and a spellchecker to our site build.
In general, it’s a good idea to include only what is absolutely
necessary in your
circle.yml config file. You don’t want your various
commands sections looking like whole shell scripts. Any time you need to
invoke a command delegate everything either to a single shell script or to a
Make command. YAML is a simple key/value markup format, not a scripting
The only tricky part of getting our setup to work was enabling CircleCI to communicate directly with our Digital Ocean droplets over SSH. Here’s how we did it:
~/.ssh/authorized_keyson both of our current Digital Ocean droplets. We now share that key with anyone who needs it via 1Password.
You’ve probably encountered this kind of flow if you’ve used AWS or DigitalOcean or similar services. The tricky part for us was making sure that both CircleCI and our engineers can perform the same tasks. After all, sometimes manual publishing and deployment tasks are still necessary, for example if CircleCI goes down at a time when our site is completely borked.
The most important thing that we learned is that you should keep your SSH
config locally in your website repo so that you can make sure that
every user (including CircleCI!) is using the same config for all
rsync commands. Our website’s SSH config is stored in a
config/ssh-config file, and the
-F flag enables us to use that config
directly when a command is invoked. Here are some examples:
# Establishes a basic SSH tunnel using the config within the repo $ ssh -F config/ssh-config reflect.io-east # Copies our local production nginx config onto a remote box $ scp -F config/ssh-config \ config/nginx/nginx-production.conf \ reflect.io-west:/etc/nginx/conf.d/reflect.io.conf
The SSH config in the website repo looks something like this:
Host reflect.io-west HostName 192.0.2.1 IdentityFile ~/.ssh/id_reflect_website User root Host reflect.io-east HostName 192.0.2.2 IdentityFile ~/.ssh/id_reflect_website User root
In order to ensure that anyone can manage the site manually, each of our
engineers who’s responsible for the site keeps the key locally at
~/.ssh/id_reflect_website. In the eyes of our Digital Ocean droplets,
CircleCI and our engineers are considered equals.
The advantages of setting up an automatic website publishing flow are very clear to us in hindsight. It took a few hours to get everything set up but it has without a doubt saved us many, many hours and afforded us peace of mind. If you use a static site generator for your website, we strongly recommend that you make the same investment. But there are some things we found out that you might be able to learn from:
git checkout master && make release.
masterpushes straight to production and
developmentpushes straight to staging, but your practices may vary. Your website is others’ portal into your organization and should be troubleshooted as thoroughly as anything else.
That’s it for now! We have more to say about our website, in particular about our Jekyll setup, how we built a local development environment for the site, and how we’ve used Jekyll plugins to do things outside of what’s offered by any static site generator that we’ve found. Stay tuned.