Building an automated website publishing pipeline using CircleCI
Deploying your website manually is often a waste of time and energy. Here’s how we used automation to forge a better path at Reflect.
I have a confession: I actually kind of enjoy manually deploying and managing
websites. I grab a cup of coffee, run
make publish or some similar command,
gaze into the CLI output, wait for static assets to be generated and
to a server thousands of miles away, and refresh the browser until I see the
new changes go live—one of life’s subtle pleasures.
But let’s face it, there’s usually no good reason to do things this way now that continuous deployment and similar automation patterns have become standard practice in our industry. We decided a while back here at Reflect to use the CI tool that we already use for everything else, CircleCI, to automate our website deployment flow, and the benefits were immediately clear.
Our website generation toolkit
Before we get into website deployment, a few words on how we build our site for context.
We use Jekyll as both our static site generator and as
the manager of our static asset pipeline.
Basically we feed a bunch of
fully built site in a
_site directory. We’ll say a lot more about this in a
Outside of this we we also use: Gulp.js for things like managing BrowserSync for local development; Rake for running our deployment scripts, which are heavily parameterized and benefit greatly from Ruby syntax (rather than Bash); and of course good old Make, which is sort of the main entry point into the project (our CircleCI jobs only ever run Make commands).
The infrastructure behind our website
So where does our site run and how do we get it there? Here’s a quick overview of our setup:
- We have two dedicated DigitalOcean droplets, one running in New York and the other running in San Francisco.
- We have both production and staging environments for the website. At the moment, both the staging and production versions of the site run side by side on the same boxes (which is fine given how light the load is for our staging site).
- All assets are served via nginx. We have separate nginx configs for the staging and production sites.
Pro tip: store your nginx configs in the same repo as the website
Initially, our nginx configs simply lived on our droplets. If we
needed to update those configs we SSHed into the boxes and made ad hoc changes.
But the downsides of this are pretty clear: it’s easy to update one box and
forget to update the other, it’s annoying to have to jump onto another box just
to know what the config even is, and so on.
So we decided to store our nginx configs in the repo alongside the site itself, which means that they’re now subject to version control and easily viewable. So how do we make sure that nginx actually uses those configs? We’ll say more about that below.
All our code lives on GitHub, which we may know a thing or two about. We use CircleCI for everything CI related at Reflect. We have numerous repos hooked up to it, we get a steady flow of info from it on a dedicated Slack channel, and we’ve been very happy with it thus far.
The CircleCI configuration for our
website lives in a
circle.yml file (as per the convention) and is currently
machine: timezone: America/Los_Angeles node: version: 7.4.0 ruby: version: 2.3.1 dependencies: override: - make setup deployment: production: branch: master commands: - make release staging: branch: development commands: - make release_staging
This config specifies that:
- Timestamps for CI output should be for the
America/Los_Angelestimezone, because that’s the one we all live in (we’re lobbying behind the scenes for an
America/Portlandtimezone…we’ll let you know how that goes)
- Node.js version 7.4.0 will be used (for things like our Gulp.js setup), while Ruby version 2.3.1 will be used for Jekyll
Prior to the build, an additional
make setupcommand will be run. That command looks like this in our
gem install bundler bundle install npm install
Our basic development pattern
Although there are always exceptions and edge cases, for the most part all projects at Reflect follow a workflow in which:
masterbranch is considered the most up to date
- all pull requests are targeted to a
developmentbranch rather than to
developmentis merged into
masteronly as part of a pull request, which enables us to review the collected result of many pull requests before we update
The website follows this flow as well. We’ve found this flow to be very
friendly to our continuous deployment patterns, as we can confidently push bold
development with the assurance that they’ll be vetted one more
time before being pushed to
master, which can mean anything from updating our
libraries in npm to updating Debian packages to, in
this case, pushing the website live.
Once the CI environment is all set up, what happens next depends on the branch to which we’re merging:
- Whenever changes are merged to
master, the production site is updated. That means that all of the following happens in CircleCI:
- Jekyll builds the site with
make releasecommand is run, which
rsyncs the contents of the
_sitefolder to both Digital Ocean droplets,
scps our nginx configuration files, and uses SSH to restart nginx and perform some basic housecleaning (like
chowning certain directories).
- Jekyll builds the site with
- Whenever changes are merged to
development, the staging version of the site is updated (this includes all of the steps for #1).
By default, CircleCI runs
make testwhenever you push to a branch that doesn’t have any behaviors associated with it. In our case, that means that
make testis run whenever we push to branches outside of
make testreturns 0 the build passes; otherwise, the build fails.
At the moment, our
make testcommand does one thing:
make build, which simply runs
jekyll build. This process exits with a 0 (success) or something else (failure). If we push anything to any branch and Jekyll can’t build it then CircleCI will let us know via Slack immediately. In the future we’d like to “test” our site more comprehensively by adding things like a linkchecker and a spellchecker to our site build.
Keep your CircleCI config minimal
In general, it’s a good idea to include only what is absolutely
necessary in your
circle.yml config file. You don’t want your various
commands sections looking like whole shell scripts. Any time you need to
invoke a command delegate everything either to a single shell script or to a
Make command. YAML is a simple key/value markup format, not a scripting
CircleCI and SSH
The only tricky part of getting our setup to work was enabling CircleCI to communicate directly with our Digital Ocean droplets over SSH. Here’s how we did it:
- We added the website’s GitHub repo to our organization’s CircleCI projects
- We created an SSH key on one of our local machines and uploaded the key to
~/.ssh/authorized_keyson both of our current Digital Ocean droplets. We now share that key with anyone who needs it via 1Password.
- We uploaded the generated key to CircleCI via the Settings page in their web UI.
You’ve probably encountered this kind of flow if you’ve used AWS or DigitalOcean or similar services. The tricky part for us was making sure that both CircleCI and our engineers can perform the same tasks. After all, sometimes manual publishing and deployment tasks are still necessary, for example if CircleCI goes down at a time when our site is completely borked.
The most important thing that we learned is that you should keep your SSH
config locally in your website repo so that you can make sure that
every user (including CircleCI!) is using the same config for all
rsync commands. Our website’s SSH config is stored in a
config/ssh-config file, and the
-F flag enables us to use that config
directly when a command is invoked. Here are some examples:
# Establishes a basic SSH tunnel using the config within the repo $ ssh -F config/ssh-config reflect.io-east # Copies our local production nginx config onto a remote box $ scp -F config/ssh-config \ config/nginx/nginx-production.conf \ reflect.io-west:/etc/nginx/conf.d/reflect.io.conf
The SSH config in the website repo looks something like this:
Host reflect.io-west HostName 192.0.2.1 IdentityFile ~/.ssh/id_reflect_website User root Host reflect.io-east HostName 192.0.2.2 IdentityFile ~/.ssh/id_reflect_website User root
In order to ensure that anyone can manage the site manually, each of our
engineers who’s responsible for the site keeps the key locally at
~/.ssh/id_reflect_website. In the eyes of our Digital Ocean droplets,
CircleCI and our engineers are considered equals.
Conclusion: make the jump, but keep a few things in mind
The advantages of setting up an automatic website publishing flow are very clear to us in hindsight. It took a few hours to get everything set up but it has without a doubt saved us many, many hours and afforded us peace of mind. If you use a static site generator for your website, we strongly recommend that you make the same investment. But there are some things we found out that you might be able to learn from:
- Make sure that the site can still be deployed manually if necessary. There’s
plenty that can go wrong, even in a heavily automated setup, and you need to
be prepared for scenarios in which your automation framework has issues. At
any point in time, several members on your team should be equipped to manually
deploy the site securely and using just a handful of (well-documented!)
commands. We have eight employees and currently half of them can deploy the
site at any time by running
git checkout master && make release.
- Run the site in multiple environments, both of which are hooked into your
automation pipeline. For us, that means that
masterpushes straight to production and
developmentpushes straight to staging, but your practices may vary. Your website is others’ portal into your organization and should be troubleshooted as thoroughly as anything else.
- Virtually everything that’s necessary for publishing the site locally should be located within the repo and subject to version control (and thus open to collaboration).
That’s it for now! We have more to say about our website, in particular about our Jekyll setup, how we built a local development environment for the site, and how we’ve used Jekyll plugins to do things outside of what’s offered by any static site generator that we’ve found. Stay tuned.