logo-color.svg Menu

How to use the Reflect Agent in your own infrastructure

Reflect’s self-hosted agent is a secure, scalable system that connects directly to your databases and communicates with the Reflect API without sharing your credentials.

Considerations

Reflect provides a cluster of agents to all users. When you set up a connection in the Reflect application, it uses our cluster. Your database credentials are securely stored in this cluster, and you must allow access to your database directly from our cluster’s external IP address.

If you do not want to share your database credentials with Reflect or expose your databases to the Internet, you have the option of running the agent as a self-hosted instance or replicated cluster. The agent communicates with the Reflect API using WebSockets; it does not require any inbound firewall changes.

When you run your own agent, you are responsible for maintaining it in your infrastructure, including any hardware or network requirements.

Downloads

The Reflect Agent is available for Debian Linux, for Ubuntu Linux, and as a Docker image.

Debian

Ubuntu

Docker

# docker run -v /your/agent/storage:/agent --name reflect-agent reflect/agent:latest

Other

If the agent isn’t available for your operating system, let us know. We’ll do our best to support it as soon as possible.

Setup

The Reflect Agent is ready to be used as soon as you install it. You can optionally configure clustering or encryption if needed.

Configuration file

When you start the agent, you can specify the path to a configuration file. By default, the agent uses the following paths:

  • If you are root, the path is /etc/reflect/reflect-agent.json.
  • For any other user, the agent uses the following ordered strategy:
    1. If the environment variable $XDG_CONFIG_HOME is defined, the path is $XDG_CONFIG_HOME/reflect/reflect-agent.json.
    2. If $HOME is defined, the path is $HOME/.config/reflect/reflect-agent.json.
    3. Otherwise, the path is /etc/reflect/reflect-agent.json.

Most options available in the Reflect Agent command-line interface can be set in the configuration file. Here is an example of a mostly complete configuration file:

{
  "data_dir": "/var/lib/reflect/agent-1",
  "key_path": "/etc/reflect/reflect-agent.key",
  "read_only": false,
  "cluster": {
    "member_name": "reflect-agent-1",
    "client_bind_addresses": ["0.0.0.0:9703"],
    "peer_bind_addresses": ["0.0.0.0:9704"],
    "join_urls": ["agent://10.10.10.10:9701"]
  },
  "storage": {
    "backend": "s3",
    "backends": {
      "s3": {"bucket": "my-bucket"}
    }
  }
}

Clustering

Agent nodes can optionally federate with each other, maintaining state and sharing credential data. Agents use the Raft consensus algorithm to coordinate activity. When planning for a Reflect Agent cluster, you should first understand the requirements of Raft, in particular how it manages quorum, the number of members required for distributed consensus:

Number of members Quorum Maximum number of failed members
1 1 0
2 2 0
3 2 1
4 3 1
5 3 2
6 4 2
7 4 3

The optimal number of cluster members is 3, 5, or 7, depending on your availability requirements. It is not recommended to run an even number of members, nor is it recommended to have more than 7 members in the same cluster.

Cluster membership must be established when first starting an agent node. Once a node has started, it cannot be moved to a different cluster. If a node is removed from a cluster, it cannot be started again.

The following options are available in the cluster section of the configuration file, assuming the public IP address of the interface of the default gateway is 10.10.10.10:

Name Type Description Default
election_timeout number The timeout, in milliseconds, for leader elections 1000
heartbeat_interval number The frequency, in milliseconds, at which to send heartbeat messages to cluster members 100
member_name string A cluster-unique name for this cluster member (Randomly generated)
client_bind_addresses string[] An array of IP address and port combinations to bind for client connections 0.0.0.0:9701
client_advertise_addresses string[] An array of IP address and port combinations corresponding to client_bind_addresses to publicly advertise to other cluster members 10.10.10.10:9701
peer_bind_addresses string[] An array of IP address and port combinations to bind for peer connections 0.0.0.0:9702
peer_advertise_addresses string[] An array of IP address and port combinations corresponding to peer_bind_addresses to publicly advertise to other cluster members 10.10.10.10:9702
join_urls string[] A list of agent client URLs to federate with  

Example

We are going to set up a two-node cluster with the following configuration:

Name OS IP Client address
alpha Debian jessie 10.10.10.10 10.10.10.10:9701
bravo Debian jessie 10.10.10.11 10.10.10.11:9701

We’ll start with alpha.

When you install the Reflect Agent using a package, it will automatically set up a single-node cluster. Let’s first disable the default instance:

# systemctl stop reflect-agent.service
# systemctl disable reflect-agent.service
Removed /etc/systemd/system/multi-user.target.wants/reflect-agent.service.

The agent uses templated systemd units to make it easy to run multiple agents on the same machine. For example, the unit reflect-agent@my-config.service will be configured by the file /etc/reflect/reflect-agent-my-config.json.

Create a new file, /etc/reflect/reflect-agent-alpha.json, with the following content:

{
  "data_dir": "/var/lib/reflect/agent-alpha",
  "cluster": {
    "member_name": "alpha"
  }
}

Create the data directory:

# mkdir /var/lib/reflect/agent-alpha
# chown reflect:reflect /var/lib/reflect/agent-alpha
# chmod 0750 /var/lib/reflect/agent-alpha

Enable and start the service:

# systemctl enable reflect-agent@alpha.service
Created symlink /etc/systemd/system/multi-user.target.wants/reflect-agent@alpha.service → /lib/systemd/system/reflect-agent@.service.
# systemctl start reflect-agent@alpha.service
# reflect-agent -c /etc/reflect/reflect-agent-alpha.json cluster members
+---+-------+--------------------------+--------+
|   | NAME  |        ADDRESSES         | LEADER |
+---+-------+--------------------------+--------+
| * | alpha | agent://10.10.10.10:9701 | Yes    |
+---+-------+--------------------------+--------+

Now let’s configure bravo. The process is essentially the same, but we’ll need to specify the URL to alpha in the configuration file /etc/reflect/reflect-agent-bravo.json:

{
  "data_dir": "/var/lib/reflect/agent-bravo",
  "cluster": {
    "member_name": "bravo",
    "join_urls": [
      "agent://10.10.10.10:9701"
    ]
  }
}

Remember, the join URLs only apply when the node is started for the very first time. They will be ignored subsequently, so if alpha is eventually removed from the cluster, bravo will continue to work without a problem (assuming it retains quorum).

Disable the default instance and set up bravo’s data directory. Then enable and start the new node:

# systemctl enable reflect-agent@bravo.service
Created symlink /etc/systemd/system/multi-user.target.wants/reflect-agent@bravo.service → /lib/systemd/system/reflect-agent@.service.
# systemctl start reflect-agent@bravo.service
# reflect-agent -c /etc/reflect/reflect-agent-bravo.json cluster members
+---+-------+--------------------------+--------+
|   | NAME  |        ADDRESSES         | LEADER |
+---+-------+--------------------------+--------+
|   | alpha | agent://10.10.10.10:9701 | Yes    |
| * | bravo | agent://10.10.10.11:9701 | No     |
+---+-------+--------------------------+--------+

Additional cluster nodes can be added in the same way.

Encryption

By default, the agent does not encrypt your credential information. (But communication between your agent and the Reflect API is always automatically encrypted using TLS.) If needed, you can enable connection credential encryption by generating a secret key and specifying the path to it in your configuration file.

The agent uses AES-256 in CBC mode with a SHA-256 HMAC.

To enable encryption, you must generate a 512-bit (64-byte) random key. For example, using dd:

# dd if=/dev/urandom of=/etc/reflect/reflect-agent.key bs=1 count=64
64+0 records in
64+0 records out
64 bytes copied, 0.000233683 s, 274 kB/s
# chown reflect /etc/reflect/reflect-agent.key
# chmod 0400 /etc/reflect/reflect-agent.key

Then set the key_path configuration option to the path to the generated key. All connections added or edited after setting the key path will be encrypted.

All members of a cluster must use the same secret key.

Login

You must authenticate to Reflect before using a self-hosted agent. This is a one-time operation: once a cluster is associated to a Reflect account, it cannot be deassociated.

# reflect-agent login
Authenticating with Reflect account credentials
E-mail address: you@example.com
Password: 
OK

You can verify the authentication attempt by listing connections. When you initially log in, your agent will request metadata about all connections in your account. Note that they are not available for use on the agent until configured.

# reflect-agent connections list
+--------------------------+--------------------------+------------+----------+----+
|           NAME           |           SLUG           |   DRIVER   | ON AGENT | UP |
+--------------------------+--------------------------+------------+----------+----+
| My Very Fancy Connection | my-very-fancy-connection | PostgreSQL | No       |    |
| Sample Connection #1     | sample-connection-1      | PostgreSQL | No       |    |
+--------------------------+--------------------------+------------+----------+----+

Backup and restore

You may want to periodically back up the data stored in your agent:

# reflect-agent backup --path /your/backups/reflect-agent-$( date -I ).backup

Only connection data is backed up, not metadata about cluster nodes. Backups are an effective way to transfer your data to a new cluster.

Only a new agent instance can be restored from a backup. If an agent has already been started or joined to a cluster, it will ignore any requests to restore data.

The first time you start an agent instance, specify the path to the backup file:

# reflect-agent serve --restore-path /your/backups/reflect-agent-2017-05-22.backup

Connection management

When using a self-hosted agent, you do not configure connections using the Reflect application. You must manage them using the reflect-agent command.

Add a connection

# reflect-agent connections add postgres \
  --name 'My Very Fancy Connection' \
  --host db.internal.example.com \
  --port 5432 \
  --user example \
  --password s3cr3t \
  --database example \
  --ssl require
Added connection "My Very Fancy Connection".

When you add a new connection, the agent will attempt to contact the data source and verify it is working. If successful, the driver and connection name are sent to the Reflect API. All other driver options are only stored on the agent.

See the list of available drivers and driver options for more information.

List connections

# reflect-agent connections list
+--------------------------+--------------------------+------------+----------+-----+
|           NAME           |           SLUG           |   DRIVER   | ON AGENT | UP  |
+--------------------------+--------------------------+------------+----------+-----+
| My Very Fancy Connection | my-very-fancy-connection | PostgreSQL | Yes      | Yes |
| Sample Connection #1     | sample-connection-1      | PostgreSQL | No       |     |
+--------------------------+--------------------------+------------+----------+-----+

Edit a connection

You can change the name or individual driver options of a given connection:

# reflect-agent connections edit my-very-fancy-connection name 'My Somewhat OK Connection'
Saved connection "My Somewhat OK Connection"
# reflect-agent connections edit my-very-fancy-connection options \
  --user readonly_example \
  --password readonly_s3cr3t
Saved connection "My Somewhat OK Connection"
# reflect-agent connections list
+---------------------------+--------------------------+------------+----------+-----+
|           NAME            |           SLUG           |   DRIVER   | ON AGENT | UP  |
+---------------------------+--------------------------+------------+----------+-----+
| My Somewhat OK Connection | my-very-fancy-connection | PostgreSQL | Yes      | Yes |
| Sample Connection #1      | sample-connection-1      | PostgreSQL | No       |     |
+---------------------------+--------------------------+------------+----------+-----+

Delete a connection

# reflect-agent connections delete my-very-fancy-connection
Deleted connection "My Somewhat OK Connection"

Drivers

Redshift

This driver supports all versions of Amazon Redshift.

Flag Description Default
host The service host name  
port The service port 5439
user A username for connecting to the database  
password The password associated with user  
database The name of the database to connect to  
connect_timeout Timeout for establishing a connection to the database, in seconds 30
ssl The SSL mode for connecting to the database require

SSL modes

Mode Description
disable No encryption
require Connections are encrypted, but no checking of the certificate chain
require-ca Connections are encrypted and certificate chain is checked
verify-full Connections are encrypted and the certificate chain and common name are checked

In most cases, you should use require or verify-ca with Redshift.

PostgreSQL

This driver supports PostgreSQL version 9.0 and higher.

Flag Description Default
host The DBMS host name localhost
port The DBMS port 5432
user A username for connecting to the database  
password The password associated with user  
database The name of the database to connect to  
connect_timeout Timeout for establishing a connection to the database, in seconds 30
ssl The SSL mode for connecting to the database disable

SSL modes

Mode Description
disable No encryption
require Connections are encrypted, but no checking of the certificate chain
require-ca Connections are encrypted and certificate chain is checked
verify-full Connections are encrypted and the certificate chain and common name are checked

MySQL

This driver supports MySQL version 4.1 and higher, MariaDB, Percona Server for MySQL, and Google CloudSQL.

Flag Description Default
host The DBMS host name localhost
port The DBMS port 3306
user A username for connecting to the database  
password The password associated with user  
database The name of the database to connect to  
timeout Timeout for establishing a connection to the database, in seconds 60

Microsoft SQL Server

This driver supports Microsoft SQL Server 2008 and higher.

Flag Description Default
host The DBMS host name localhost
port The DBMS port 1433
user A username for connecting to the database sa
password The password associated with user  
database The name of the database to connect to  
timeout The timeout in seconds for the connection attempt 30
ssl The SSL mode for connecting to the database verify

SSL modes

Mode Description
disable No encryption
no-verify Connections are encrypted, but no checking of the the server certificate
verify Connections are encrypted and the server certificate is checked

SQLite

This driver supports SQLite version 3.0 and higher.

Flag Description
source_url An HTTP(S) URL to download the SQLite database from
refresh_interval The frequency in seconds to refresh this connection’s data from the source URL

CSV

Flag Description
source_url An HTTP(S) URL to download the CSV file from
time_format A time format to use when parsing
refresh_interval The frequency in seconds to refresh this connection’s data from the source URL

HTTP

Contact us for more information and an API specification.

Cluster management

The setup section describes the process for configuring a cluster.

List members

# reflect-agent cluster members
+---+---------------------+--------------------------+--------+
|   |        NAME         |        ADDRESSES         | LEADER |
+---+---------------------+--------------------------+--------+
|   | reflect-agent-tno-1 | agent://10.10.10.10:9701 | Yes    |
|   | reflect-agent-tno-2 | agent://10.10.10.11:9701 | No     |
| * | reflect-agent-tno-3 | agent://10.10.10.12:9701 | No     |
+---+---------------------+--------------------------+--------+

Remove a member

# reflect-agent cluster remove-member reflect-agent-tno-2
+---+---------------------+--------------------------+--------+
|   |        NAME         |        ADDRESSES         | LEADER |
+---+---------------------+--------------------------+--------+
|   | reflect-agent-tno-1 | agent://10.10.10.10:9701 | Yes    |
| * | reflect-agent-tno-3 | agent://10.10.10.12:9701 | No     |
+---+---------------------+--------------------------+--------+