Virtual Event
Join us for the next HashiConf Digital October 12-15, 2020 Register for Free

Secure Your Datacenters with Agent Authentication

Update Consul Agents to Securly Communicate with TLS

Securing your datacenter with TLS encryption for RPC and consensus communication is a prerequisite of the Security Model. This tutorial provides the necessary steps to update your existing datacenter for production ready TLS with zero-downtime.

You must complete the following eight-steps successfully, completing each step before starting the next, to update agents to communicate over TLS.

  1. Generate server certificates. Generate a client certificate if you're using the operator method.
  2. Distribute certificates, including the CA and private key files to every agent.
  3. Configure the servers, allowing encrypted traffic.
  4. Restart each server, one at a time.
  5. Configure the clients to communicate only with TLS
  6. Restart each client.
  7. Reconfigure servers to only allow TLS communication.
  8. Reload each server, one at a time.

»Prerequisites

Before starting this tutorial, you should have an existing Consul datacenter with running server and client agents.

You will need to generate your certificates before starting this tutorial. Follow the steps in the Secure Agent Communication with TLS Encryption tutorial to use the built-in CA or the Secure Agent Communication with Existing Certificate Authority tutorial to use OpenSSL.

Finally, you should decide which certificate distribution method works for you. There are two methods for distributing client certificates; operator or auto encryption. Both methods are described in the "Secure Agent Communication with TLS Encryption" tutorial.

The example configurations for this tutorial are in JSON. You can copy each one of the examples and add them to your existing JSON configuration file.

»Distribute the certificates

Before configuring the agents, you will need to distribute the certificate and key files.

»Server certificates

Distribute the following certificates and key files to each of your servers.

Note, the CA certificate may be named differently if you are not using Consul's built-in CA. The server certificate and key file should be unique for each server.

»Client certificate for operator method

Distribute the client certificate and key file to all your clients. All clients should have the following certificates and key file.

»Configure the servers

Now that you have created the certificates you can enable TLS on your agents, starting with the servers. For both client certificate distribution methods, operator and auto encryption, you must set the following options to false.

These options must be set to false to allow existing communication to continue uninterrupted. This will require a rolling restart on all the servers. Once the clients are configured, you can then enable verify_incoming, verify_outgoing, and verify_server_hostname on the servers. In addition to the verify options, you will set the certificates and key file the agent should use.

Below are the configurations for both methods.

»Operator method configuration

{
  "verify_incoming": false,
  "verify_outgoing": false,
  "verify_server_hostname": false,
  "ca_file": "consul-agent-ca.pem",
  "cert_file": "dc1-server-consul-0.pem",
  "key_file": "dc1-server-consul-0-key.pem"
}

»Auto encryption method configuration

For the auto encryption method, you will need to set auto_encrypt to true.

{
  "verify_incoming": false,
  "verify_outgoing": false,
  "verify_server_hostname": false,
  "ca_file": "consul-agent-ca.pem",
  "cert_file": "dc1-server-consul-0.pem",
  "key_file": "dc1-server-consul-0-key.pem",
  "auto_encrypt": {
    "allow_tls": true
  }
}

Setting the verify options to false on the servers allows you to update all the agents without any downtime since authenticity verification is not enforced. Note, leaving the verify settings as false will create an insecure configuration.

»Restart the Consul servers

After configuring the servers, for either the operator or auto encryption client certificate distribution method, you can initiate a rolling restart of the servers.

Be sure to restart one server at a time, ensuring it's functioning properly before restarting the next. The leader should be restarted last, to avoid downtime.

$ systemctl restart consul

»Configure the clients

Now that you have enabled TLS communication on the servers and distributed the certificates, you can configure the clients.

»Operator method

For the operator method, you will configure the clients to use the operator distributed certificates and key file.

{
  "verify_incoming": true,
  "verify_outgoing": true,
  "verify_server_hostname": true,
  "ca_file": "consul-agent-ca.pem",
  "cert_file": "dc1-server-consul-0.pem",
  "key_file": "dc1-server-consul-0-key.pem"
}

»Auto encryption method

For the auto encryption method, you will enable auto_encrypt.

{
  "verify_incoming": false,
  "verify_outgoing": true,
  "verify_server_hostname": true,
  "ca_file": "consul-agent-ca.pem",
  "auto_encrypt": {
    "tls": true
  }
}

You do not need to distribute client certificate using the auto encryption method. Consul and Consul Connect will manage certificate distribution for you.

»Restart the Consul clients

Finally, restart the clients. You will only need to restart the clients once.

$ systemctl restart consul

»Reconfigure and reload the servers

»Operator method

For the operator method, you will update verify_incoming, verify_outgoing, and verify_server_hostname to true.

{
  "verify_incoming": true,
  "verify_outgoing": true,
  "verify_server_hostname": true,
  "ca_file": "consul-agent-ca.pem",
  "cert_file": "dc1-server-consul-0.pem",
  "key_file": "dc1-server-consul-0-key.pem"
}

»Auto encryption method

For the auto encryption method, you will update verify_incoming, verify_outgoing, and verify_server_hostname to true

{
  "verify_incoming": true,
  "verify_outgoing": true,
  "verify_server_hostname": true,
  "ca_file": "consul-agent-ca.pem",
  "cert_file": "dc1-server-consul-0.pem",
  "key_file": "dc1-server-consul-0-key.pem",
  "auto_encrypt": {
    "allow_tls": true
  }
}

»Reload the Consul servers

After a final Consul agent reload, your agents will only communicating with TLS.

$ consul reload

»Next steps

In this tutorial, you configured your Consul agents to communicate over TLS. The other prerequisites for a secure Consul deployment are gossip encryption and ACLs with the agents configured with default deny.