The new self-hosted Reflect Agent

You can now provide data to Reflect’s reporting and visualization API without sharing your database connection info.

The Reflect data visualization and reporting API is a powerful tool that enables you to use data from a variety of sources and easily produce and distribute data reports and visualizations.

Crucial to the Reflect platform is the ability of our API to securely connect to a variety of sources—without requiring ETL or a complicated ingestion API—using a process that we call the Reflect Agent.

Reflect offers a hosted cluster of Agents for all users, paid and free tier. Previously, using Reflect meant using only our hosted Agents. But now that’s changing, as you can now run your own self-hosted Reflect Agent on your own infrastructure and your own terms.

Supported data sources

The Reflect Agent currently enables you to securely connect to four databases:

We’ll be expanding the number of supported databases very soon, so stay tuned for announcements here on the blog or on Twitter or Facebook.

Why run a self-hosted Reflect Agent?

The advantage of using a self-hosted Reflect Agent is that it enables you to securely share your data with the Reflect API without needing to provide your database connection information with Reflect. Self-hosted agents store all database information encrypted and in an embedded database. If you have security requirements that prevent you from sharing database credentials with a third party (even read-only credentials) then you should explore the Reflect Agent as a solution.

How the Reflect Agent works

The Reflect Agent acts as a mediary between two things:

  1. Your data sources (we currently support Amazon Redshift, PostgreSQL, and MySQL, with more on the way soon)
  2. The Reflect API, which provides our public-facing data reporting and visualization interface (essentially our core product)

When you run your own Reflect Agent, our API makes requests only to your agent (or cluster of agents), not directly to your databases.

Installing and running

Check out our documentation

In this blog post we’ll provide just a very basic overview of how the Reflect Agent works and how you can install and use it. For a full treatment, see our Agent doc.

The Reflect Agent can be installed and run in two ways:

We currently offer the Reflect Agent for the following Linux distributions:

Just download the package for your distro, go through some basic setup steps, and register your agent with with our API.

If you want to run the Reflect Agent as a Docker container, you can simply pull the image from Docker Hub and start it:

$ docker pull reflect/agent:latest
$ docker run \
  -v /opt/reflect/agent:/agent # Specify a storage volume
  --name=reflect-agent         # Provide a name for the container
  --net=host \                 # Run the container in "host" mode (required)
  reflect/agent:latest         # The Reflect Agent Docker image

The clustered Reflect Agent

Running the Reflect Agent on a single server should be enough to handle most development and staging workloads, but for production workloads we recommend running multiple agents as a cluster.

When running in a cluster, Reflect agents rely on an implementation of the Raft consensus protocol to coordinate cluster state across nodes. This enables a cluster of agents to provide higher fault tolerance than running a single agent.

Conclusion: same Reflect platform, now with more security options

Whether you’re using a self-hosted Reflect Agent (or cluster of agents) or using the agents that Reflect provides, you can always connect to your databases securely. But if you need to go an extra step and use Reflect without sharing your database connection information or exposing your databases to the public internet, running a self-hosted Reflect Agent is an option that you should absolutely bear in mind.