Day 1: Security and Network Operations

Secure Agent Communication with TLS Encryption

Securing your datacenter with TLS encryption is an important step for production deployments. TLS configuration is a prerequisite of our Security Model. Correctly configuring TLS can be a complex process, especially given the wide range of deployment methodologies. This guide will provide you with a production ready TLS configuration for RPC, Consensus, and HTTP communication.

Consul supports using TLS to verify the authenticity of servers and clients. To enable TLS, Consul requires that all servers have certificates that are signed by a single Certificate Authority(CA). Clients should also have certificates that are authenticated with the same CA.

This guide is for new Consul deployments and has the following sections:

  1. Creating Certificates
  2. Configuring Agents
  3. Configuring the Consul CLI for HTTPS
  4. Configuring the Consul UI for HTTPS

Please review the call outs for existing Consul deployments.

This guide is structured in way that you build knowledge with every step. It is recommended to read the whole guide before starting with the actual work, because you can save time if you are aware of some of the more advanced things in section 3 and 4.

More advanced topics like key management and rotation are not covered by this guide. Vault is the suggested solution for key generation and management.

Creating Certificates

The first step to configuring TLS for Consul is generating certificates. In order to prevent unauthorized datacenter access, Consul requires all certificates be signed by the same Certificate Authority (CA). This should be a private CA and not a public one like Let's Encrypt as any certificate signed by this CA will be allowed to communicate with the datacenter.

You can create the CA and certificates before starting Consul, as long as you have the Consul binary installed in your path. We recommend creating the CA on a stable server that will always be part of your datacenter.

Step 1: Create a Certificate Authority

There are a variety of tools for managing your own CA, like the PKI secret backend in Vault, but for the sake of simplicity this guide will use Consul's built-in TLS helpers to create a basic CA. You will only need to create one CA for the datacenter. You should generate all certificates on the same server that is used to create the CA.

$ consul tls ca create
==> Saved consul-agent-ca.pem
==> Saved consul-agent-ca-key.pem

The CA certificate, consul-agent-ca.pem, contains the public key necessary to validate Consul certificates and therefore must be distributed to every node that runs a consul agent.

The CA key, consul-agent-ca-key.pem, will be used to sign certificates for Consul nodes and must be kept private. Possession of this key allows anyone to run Consul as a trusted server and access all Consul data, including ACL tokens.

Step 2: Create Individual Server Certificates

Create a server certificate for datacenter dc1 and domain consul, if your datacenter or domain is different please use the appropriate flags.

$ consul tls cert create -server
==> WARNING: Server Certificates grants authority to become a
    server and access all state in the cluster including root keys
    and all ACL tokens. Do not distribute them to production hosts
    that are not server nodes. Store them as securely as CA keys.
==> Using consul-agent-ca.pem and consul-agent-ca-key.pem
==> Saved dc1-server-consul-0.pem
==> Saved dc1-server-consul-0-key.pem

Repeat this process on the same server where you created the CA, until there is an individual certificate for each server. The command can be called over and over again, it will automatically increase the certificate and key numbers. You will need to distribute the certificates to the servers.

In order to authenticate Consul servers, servers are provided with a special certificate - one that contains server.dc1.consul in the Subject Alternative Name. If you enable verify_server_hostname, only agents that provide such certificate are allowed to boot as a server. Without verify_server_hostname = true an attacker could compromise a Consul client agent and restart the agent as a server in order to get access to all the data in your datacenter! This is why server certificates are special, and only servers should have them provisioned.

Step 3: Client Certificates

Next, create the client certificates. In Consul 1.5.2, you can use an alternative process to automatically distribute certificates to the clients. To enable this new feature, set auto_encrypt.

You can continue to generate client certificates with consul tls cert create -client and manually distribute certificates. This existing workflow is still necessary for datacenters that need to be highly secured.

Configuring Consul Agents: Servers and Clients

The example configurations from this, as well as the following sections, are in json. You can copy each one of the examples in its own file in a directory (-config-dir) from where consul will load all the configuration. This is just one way to do it, you can also put them all into one file if you prefer that.

Now that you have created the certificates you need to enable TLS in your datacenter. The next steps shows you how to configure TLS for a new Consul deployment. If you have an existing deploying in production without TLS, please notice the call-out boxes in this guide for the steps needed to introduce TLS without downtime.

Step 1: Setup Consul Servers with Certificates

This step describes how to setup one of your consul servers, you want to make sure to repeat the process for the other ones as well with their individual certificates.

The following files need to be copied to your Consul server:

Here is an example agent TLS configuration for Consul servers which mentions the copied files:

{
  "verify_incoming": true,
  "verify_outgoing": true,
  "verify_server_hostname": true,
  "auto_encrypt": {
    "allow_tls": true
  },
  "ca_file": "consul-agent-ca.pem",
  "cert_file": "dc1-server-consul-0.pem",
  "key_file": "dc1-server-consul-0-key.pem",
  "ports": {
    "http": -1,
    "https": 8501
  }
}

TLS can be used to verify the authenticity of the servers with verify_outgoing and verify_server_hostname. It can also optionally verify client certificates when using verify_incoming. Note, when auto_enrypt is enabled, verify_outgoing is also enabled by default. The clients will inherit the verify_server_hostname value from the servers.

Review the docs for specifics.

This configuration disables the HTTP port to make sure there is only encrypted communication. Existing clients that are not yet prepared to talk HTTPS won't be able to connect afterwards. This also affects built-in tooling like consul members and the UI. The next sections will demonstrate how to setup secure access.

After the Consul agent is started, it should be only using TLS encryption for communication.

Step 2: Setup Consul Clients with Certificates

Now enable the client configuration to automatically get the certificates from the server. They are in memory and not persisted.

Here is an example agent TLS configuration.

{
  "verify_incoming": true,
  "ca_file": "consul-agent-ca.pem",
  "auto_encrypt": {
    "tls": true
  },
  "ports": {
    "http": -1,
    "https": 8501
  }
}

Note, the clients are inheriting the verify_server_hostname parameter from the server.

This configuration disables the HTTP port to make sure there is only encrypted communication. Existing clients that are not yet prepared to talk HTTPS won't be able to connect afterwards. This also affects built-in tooling like consul members and the UI. The next section will demonstrate how to setup secure access.

After a Consul agent restart, your agents should be only talking TLS.

Configuring the Consul CLI for HTTPS

If your datacenter is configured to only communicate via HTTPS, you will need to create additional certificates in order to be able to continue to access the API and the UI:

$ consul tls cert create -cli
==> Using consul-agent-ca.pem and consul-agent-ca-key.pem
==> Saved dc1-cli-consul-0.pem
==> Saved dc1-cli-consul-0-key.pem

If you are trying to get members of you dataceter, the CLI will return an error:

$ consul members
Error retrieving members:
  Get http://127.0.0.1:8500/v1/agent/members?segment=_all:
  dial tcp 127.0.0.1:8500: connect: connection refused
$ consul members -http-addr="https://localhost:8501"
Error retrieving members:
  Get https://localhost:8501/v1/agent/members?segment=_all:
  x509: certificate signed by unknown authority

But it will work again if you provide the certificates you provided:

$ consul members -ca-file=consul-agent-ca.pem -client-cert=dc1-cli-consul-0.pem \
  -client-key=dc1-cli-consul-0-key.pem -http-addr="https://localhost:8501"
  Node     Address         Status  Type    Build     Protocol  DC   Segment
  ...

This process can be cumbersome to type each time, so the Consul CLI also searches environment variables for default values. Set the following environment variables in your shell:

$ export CONSUL_HTTP_ADDR=https://localhost:8501
$ export CONSUL_CACERT=consul-agent-ca.pem
$ export CONSUL_CLIENT_CERT=dc1-cli-consul-0.pem
$ export CONSUL_CLIENT_KEY=dc1-cli-consul-0-key.pem
  • CONSUL_HTTP_ADDR is the URL of the Consul agent and sets the default for -http-addr.
  • CONSUL_CACERT is the location of your CA certificate and sets the default for -ca-file.
  • CONSUL_CLIENT_CERT is the location of your CLI certificate and sets the default for -client-cert.
  • CONSUL_CLIENT_KEY is the location of your CLI key and sets the default for -client-key.

After these environment variables are correctly configured, the CLI will respond as expected.

Note on SANs for Server and Client Certificates

Using localhost and 127.0.0.1 as Subject Alternative Names in server and client certificates allows tools like curl to be able to communicate with Consul's HTTPS API when run on the same host. Other SANs may be added during server/client certificates creation with -additional-dnsname to allow remote HTTPS requests from other hosts.

Configuring the Consul UI for HTTPS

If your servers and clients are configured now like above, you won't be able to access the builtin UI anymore. We recommend that you pick one (or two for availability) Consul agent you want to run the UI on and follow the instructions to get the UI up and running again.

Step 1: Which interface to bind to?

Depending on your setup you might need to change to which interface you are binding because thats 127.0.0.1 by default for the UI. Either via the addresses.https or client_addr option which also impacts the DNS server. The Consul UI is unprotected which means you need to put some auth in front of it if you want to make it publicly available!

Binding to 0.0.0.0 should work:

{
  "ui": true,
  "client_addr": "0.0.0.0",
  "enable_script_checks": false,
  "disable_remote_exec": true
}

Step 2: verify_incoming_rpc

Your Consul agent will deny the connection straight away because verify_incoming is enabled.

If set to true, Consul requires that all incoming connections make use of TLS and that the client provides a certificate signed by a Certificate Authority from the ca_file or ca_path. This applies to both server RPC and to the HTTPS API.

Since the browser doesn't present a certificate signed by our CA, you cannot access the UI. If you curl your HTTPS UI the following happens:

$ curl https://localhost:8501/ui/ -k -I
curl: (35) error:14094412:SSL routines:SSL3_READ_BYTES:sslv3 alert bad certificate

This is the Consul HTTPS server denying your connection because you are not presenting a client certificate signed by your Consul CA. There is a combination of options however that allows us to keep using verify_incoming for RPC, but not for HTTPS:

{
  "verify_incoming": false,
  "verify_incoming_rpc": true
}

With the new configuration, it should work:

$ curl https://localhost:8501/ui/ -k -I
HTTP/2 200
...

Step 3: Subject Alternative Name

This step will take care of setting up the domain you want to use to access the Consul UI. Unless you only need to access the UI over localhost or 127.0.0.1 you will need to go complete this step.

$ curl https://consul.example.com:8501/ui/ \
  --resolve 'consul.example.com:8501:127.0.0.1' \
  --cacert consul-agent-ca.pem
curl: (51) SSL: no alternative certificate subject name matches target host name 'consul.example.com'
...

The above command simulates a request a browser is making when you are trying to use the domain consul.example.com to access your UI. The problem this time is that your domain is not in Subject Alternative Name of the Certificate. We can fix that by creating a certificate that has our domain:

$ consul tls cert create -server -additional-dnsname consul.example.com
...

And if you put your new cert into the configuration of the agent you picked to serve the UI and restart Consul, it works now:

$ curl https://consul.example.com:8501/ui/ \
  --resolve 'consul.example.com:8501:127.0.0.1' \
  --cacert consul-agent-ca.pem -I
HTTP/2 200
...

Step 4: Trust the Consul CA

So far we have provided curl with our CA so that it can verify the connection, but if we stop doing that it will complain and so will our browser if you visit your UI on https://consul.example.com:

$ curl https://consul.example.com:8501/ui/ \
  --resolve 'consul.example.com:8501:127.0.0.1'
curl: (60) SSL certificate problem: unable to get local issuer certificate
...

You can fix that by trusting your Consul CA (consul-agent-ca.pem) on your machine, please use Google to find out how to do that on your OS.

$ curl https://consul.example.com:8501/ui/ \
  --resolve 'consul.example.com:8501:127.0.0.1' -I
HTTP/2 200
...

Summary

When you have completed this guide, your Consul datacenter will have TLS enabled and will encrypt all RPC, consensus, and HTTP traffic. Note, with TLS encryption configured, you will need to reference the CA cert, client cert file, and client key file to complete any API/CLI operation since they are HTTP traffic.

The other prerequisites for a secure Consul deployment are: