Virtual Event
Join us for the next HashiConf Digital October 12-15, 2020 Register for Free

Day 1: Deploy Your First Datacenter

Deployment Guide

This deployment guide covers the steps required to install and configure a secure HashiCorp Consul datacenter as defined in the Consul Reference Architecture.

This deployment guide is designed to work in combination with the Consul Reference Architecture. Although not a strict requirement to follow the Consul Reference Architecture, it is highly recommended that you are familiar with the overall architecture design; for example installing Consul server agents on multiple physical or virtual (with correct anti-affinity) hosts for high-availability.

To provide a highly-available single datacenter architecture, we recommend Consul server agents be deployed to more than one host, as shown in the Consul Reference Architecture.

Reference Diagram

These setup steps should be completed on all Consul hosts.

  1. Download Consul
  2. Install Consul
  3. Prepare the security credentials
  4. Configure Consul agents
  5. Configure the Consul process
  6. Start the Consul service
  7. Setup Consul environment variables
  8. Bootstrap the ACL System
  9. Apply Enterprise license
  10. Next steps

These instructions are for installing and configuring Consul on Linux hosts running the systemd system and service manager.

»Download Consul

Precompiled Consul binaries are available for download at https://releases.hashicorp.com/consul/ and Consul Enterprise binaries are available for download by following the instructions made available to HashiCorp Consul customers.

You should perform checksum verification of the zip packages using the SHA256SUMS and SHA256SUMS.sig files available for the specific release version. HashiCorp provides a guide on checksum verification for precompiled binaries.

First, export a couple environment variables to specify the Consul download base URL and preferred Consul version for convenience and concise commands. Then use curl to download the package and SHA256 summary files.

$ export CONSUL_VERSION="1.8.0"
$ export CONSUL_URL="https://releases.hashicorp.com/consul"
curl --silent --remote-name \
  ${CONSUL_URL}/${CONSUL_VERSION}/consul_${CONSUL_VERSION}_linux_amd64.zip
curl --silent --remote-name \
  ${CONSUL_URL}/${CONSUL_VERSION}/consul_${CONSUL_VERSION}_SHA256SUMS
curl --silent --remote-name \
  ${CONSUL_URL}/${CONSUL_VERSION}/consul_${CONSUL_VERSION}_SHA256SUMS.sig

»Install Consul

Unzip the downloaded package and move the consul binary to /usr/local/bin/. Check consul is available on the system path.

$ unzip consul_${CONSUL_VERSION}_linux_amd64.zip
$ sudo chown root:root consul
$ sudo mv consul /usr/local/bin/
$ consul --version

The consul command features opt-in autocompletion for flags, subcommands, and arguments (where supported). Enable autocompletion.

$ consul -autocomplete-install
$ complete -C /usr/local/bin/consul consul

Create a unique, non-privileged system user to run Consul and create its data directory.

$ sudo useradd --system --home /etc/consul.d --shell /bin/false consul
$ sudo mkdir --parents /opt/consul
$ sudo chown --recursive consul:consul /opt/consul

»Prepare the security credentials

In this section, you will create the encryption key and certificates needed to secure your Consul agents.

All commands in this section can be completed on any system with the Consul binary installed. The outputs and artifacts from these commands should be archived in a secure location for future reference.

»Generate the gossip encryption key

Gossip is encrypted with a symmetric key, since gossip between nodes is done over UDP. All agents must have the same encryption key.

You can create the encryption key via the Consul CLI even though no Consul agents are running yet. Generate the encryption key.

$ consul keygen

The encryption key will be plain text output:

$ consul keygen
qDOPBEr+/oUVeOFQOnVypxwDaHzLrD+lvjo5vCEBbZ0=

Note: You will need to add the newly generated key to the encrypt option in the server configuration on all Consul agents. Save your key in a safe location. You will need to reference the key throughout the installation.

For in-depth information about setting up gossip encryption, review the Secure Gossip Communication with Encryption tutorial.

»Generate TLS certificates for RPC encryption

Consul can use TLS to verify the authenticity of servers and clients. To enable TLS, Consul requires that all agents have certificates signed by a single Certificate Authority (CA).

In this tutorial, you will use Consul to create a CA for your certificates. For in-depth information about setting up TLS certificates, review the Secure Consul Agent Communication with TLS Encryption tutorial. To use an existing CA, review the Secure Consul Agent Communication with TLS Encryption and OpenSSL Certificates tutorial.

»Create the Certificate Authority

Start by creating the CA on your admin instance, using the Consul CLI.

$ consul tls ca create
==> Saved consul-agent-ca.pem
==> Saved consul-agent-ca-key.pem

»Create the certificates

Next create a set of certificates, one for each Consul agent. You will need to select a name for your primary datacenter now, so that the certificates are named properly.

First, for your Consul servers, use the following command to create a certificate for each server. The file name increments automatically.

$ consul tls cert create -server -dc <dc_name>

Below is an example for datacenter "dc1".

$ consul tls cert create -server -dc dc1
==> WARNING: Server Certificates grants authority to become a
    server and access all state in the cluster including root keys
    and all ACL tokens. Do not distribute them to production hosts
    that are not server nodes. Store them as securely as CA keys.
==> Using consul-agent-ca.pem and consul-agent-ca-key.pem
==> Saved dc1-server-consul-0.pem
==> Saved dc1-server-consul-0-key.pem

Use the following command with the -client flag to create client certificates. The file name increments automatically.

$ consul tls cert create -client -dc <dc_name>

Below is an example for datacenter "dc1":

$ consul tls cert create -client -dc dc1
==> Using consul-agent-ca.pem and consul-agent-ca-key.pem
==> Saved dc1-client-consul-0.pem
==> Saved dc1-client-consul-0-key.pem

»Distribute the certificates to agents

You must distribute the CA certificate, consul-agent-ca.pem, to each of the Consul agents as well as the agent specific certificate and private key.

Below is an SCP example which will send the CA certificate, agent certificate and private key to the IP address you specify, and put it into the /etc/consul.d/ directory.

$ scp consul-agent-ca.pem <dc-name>-<server/client>-consul-<cert-number>.pem <dc-name>-<server/client>-consul-<cert-number>-key.pem <USER>@<PUBLIC_IP>:/etc/consul.d/

»Configure Consul agents

Consul uses documented reasonable defaults so only non-default values must be set in the configuration file. Configuration can be read from multiple files and is loaded in lexical order. Check the full description for more information about configuration loading and merge semantics.

Consul server agents typically require a superset of configuration required by Consul client agents. You will specify common configuration used by all Consul agents in consul.hcl and server specific configuration in server.hcl.

Create a configuration file at /etc/consul.d/consul.hcl:

$ sudo mkdir --parents /etc/consul.d
$ sudo touch /etc/consul.d/consul.hcl
$ sudo chown --recursive consul:consul /etc/consul.d
$ sudo chmod 640 /etc/consul.d/consul.hcl

Add this configuration to the consul.hcl configuration file:

datacenter = "dc1"
data_dir = "/opt/consul"
encrypt = "qDOPBEr+/oUVeOFQOnVypxwDaHzLrD+lvjo5vCEBbZ0="
ca_file = "/etc/consul.d/consul-agent-ca.pem"
cert_file = "/etc/consul.d/dc1-server-consul-0.pem"
key_file = "/etc/consul.d/dc1-server-consul-0-key.pem"
verify_incoming = true
verify_outgoing = true
verify_server_hostname = true
  • datacenter - The datacenter in which the agent is running.
  • data_dir - The data directory for the agent to store state.
  • encrypt - Specifies the gossip encryption key to use for Consul network traffic.
  • ca_file - Specifies the path to the CA public certificate file.
  • cert_file - Specifies the path to the agents public certificate file.
  • key_file - Specifies the path to the agents certificate private key file.
  • verify_incoming - If set to true, Consul requires all incoming connections to use TLS.
  • verify_outgoing - If set to true, Consul requires that all outgoing connections from this agent use TLS.
  • verify_server_hostname - If set to true, Consul verifies for all outgoing TLS connections that the TLS certificate presented by the servers matches server.<datacenter>.<domain> hostname.

»Datacenter auto-join

The retry_join parameter allows you to configure all Consul agents to automatically form a datacenter using a common Consul server accessed via DNS address, IP address or using Cloud Auto-join. This removes the need to manually join the Consul datacenter nodes together.

Add the retry_join parameter to the consul.hcl configuration file:

retry_join = ["172.16.0.11"]
  • retry_join - Address of another agent to join upon starting up.

»Enable Consul ACLs

Consul uses Access Control Lists (ACLs) to secure the UI, API, CLI, and Consul catalog including service and agent registration. When securing your datacenter you should configure the ACLs first.

The Secure Consul with Access Control Lists (ACLs) tutorial provides instructions on configuring and enabling ACLs on new agents.

Add the ACL configuration to the consul.hcl configuration file and choose a default policy of "allow" (allow all traffic unless explicitly denied) or "deny" (deny all traffic unless explicitly allowed).

acl = {
  enabled = true
  default_policy = "allow"
  enable_token_persistence = true
}
  • acl - ACL stanza
  • enabled - Enables ACLs
  • default_policy - The default policy controls the behavior of a token when there is no matching rule.
  • enable_token_persistence - When true tokens set using the API will be persisted to disk and reloaded when an agent restarts.

»Performance stanza

The performance stanza allows tuning the performance of different subsystems in Consul.

Add the performance configuration to the consul.hcl configuration file:

performance {
  raft_multiplier = 1
}
  • raft_multiplier - An integer multiplier used by Consul servers to scale key Raft timing parameters. Setting this to a value of 1 will configure Raft to its highest-performance mode, equivalent to the default timing of Consul prior to 0.7, and is recommended for production Consul servers.

For more information on Raft tuning and the raft_multiplier setting, check the server performance documentation.

»Telemetry stanza

The telemetry stanza specifies various configurations for Consul to publish metrics to upstream systems.

If you decide to configure Consul to publish telemetry data, you should complete the Monitor Consul Datacenter Health tutorial at the end of the Day 1 learning path.

»Audit

The audit stanza, added in Consul 1.8, allow users to enable auditing and configure a sink and filters for their audit logs.

Add the audit configuration to the consul.hcl configuration file:

audit {
  enabled = true
  sink "My sink" {
    type   = "file"
    format = "json"
    path   = "data/audit/audit.json"
    delivery_guarantee = "best-effort"
    rotate_duration = "24h"
    rotate_max_files = 15
    rotate_bytes = 25165824
  }
}

Configuration options for the audit stanza can be found here.

»Consul server configuration

Create a configuration file at /etc/consul.d/server.hcl:

$ sudo touch /etc/consul.d/server.hcl
$ sudo chown --recursive consul:consul /etc/consul.d
$ sudo chmod 640 /etc/consul.d/server.hcl

Add this configuration to the server.hcl configuration file.

server = true
bootstrap_expect = 3
  • server - This flag is used to control if an agent is in server or client mode.
  • bootstrap-expect - This flag provides the number of expected servers in the datacenter. Either this value should not be provided or the value should be consistent across all servers in the datacenter.

»Consul UI

Consul features a web-based user interface, allowing you to easily view all services, nodes, intentions and more using a graphical user interface, rather than the CLI or API.

Optionally, add the UI configuration to the server.hcl configuration file to enable the Consul UI:

ui = true
client_addr = "0.0.0.0"
  • ui - Enables the build-in web UI.
  • client_addr - The address to which Consul will bind client interfaces, including the HTTP and DNS servers. By default, this is "127.0.0.1", allowing only loopback connections.

»Configure client configuration

Consul client agents typically require a subset of configuration required by Consul server agents. All Consul clients can use the consul.hcl file created earlier in the configuring the Consul agents section. If you have added host-specific configuration such as identifiers, you will need to set these individually.

»Configure the Consul process

»Configure systemd

Systemd uses documented reasonable defaults so only non-default values must be set in the configuration file.

Create a Consul service file at /etc/systemd/system/consul.service.

$ sudo touch /etc/systemd/system/consul.service

Add this configuration to the Consul service file.

[Unit]
Description="HashiCorp Consul - A service mesh solution"
Documentation=https://www.consul.io/
Requires=network-online.target
After=network-online.target
ConditionFileNotEmpty=/etc/consul.d/consul.hcl

[Service]
Type=notify
User=consul
Group=consul
ExecStart=/usr/local/bin/consul agent -config-dir=/etc/consul.d/
ExecReload=/usr/local/bin/consul reload
ExecStop=/usr/local/bin/consul leave
KillMode=process
Restart=on-failure
LimitNOFILE=65536

[Install]
WantedBy=multi-user.target

The following parameters are set for the [Unit] stanza:

  • Description - Free-form string describing the consul service
  • Documentation - Link to the consul documentation
  • Requires - Configure a requirement dependency on the network service
  • After - Configure an ordering dependency on the network service being started before the consul service
  • ConditionFileNotEmpty - Check for a non-zero sized configuration file before consul is started

The following parameters are set for the [Service] stanza:

  • Type - Set to notify so systemd will not attempt to start any dependent services until after Consul sends the notify signal
  • User, Group - Run Consul as the consul user
  • ExecStart - Start Consul with the agent argument and path to the configuration file
  • ExecReload - Send Consul a reload signal to trigger a configuration reload in Consul
  • ExecStop - Stop Consul with the leave argument to allow for a graceful leave from the datacenter
  • KillMode - Treat consul as a single process
  • Restart - Restart consul unless it returned a clean exit code
  • LimitNOFILE - Set an increased limit for file descriptors

The following parameters are set for the [Install] stanza:

  • WantedBy - Creates a weak dependency on Consul being started by the multi-user run level

»Start the Consul service

Check that your configuration file is valid, with the Consul CLI validate command.

$ consul validate /etc/consul.d/consul.hcl
Configuration is valid!

Enable and start Consul using the systemctl command responsible for controlling systemd managed services. Check the status of the consul service using systemctl.

$ sudo systemctl enable consul
$ sudo systemctl start consul
$ sudo systemctl status consul

The output should indicate the service is running.

$ sudo systemctl status consul
● consul.service - Consul
   Loaded: loaded (/etc/systemd/system/consul.service; disabled; vendor preset: enabled)
   Active: active (running) since Fri 2019-09-06 18:44:55 UTC; 4s ago
     Docs: https://www.consul.io/docs/
 Main PID: 13244 (consul)
    Tasks: 10
   Memory: 10.2M
      CPU: 86ms
   CGroup: /system.slice/consul.service
           └─13244 /usr/local/bin/consul agent -config-dir=/etc/consul.d

Sep 06 18:44:56 consul201-bison.node.consul consul[13244]:     2019/09/06 18:44:56 [INFO] a
Sep 06 18:44:56 consul201-bison.node.consul consul[13244]:     2019/09/06 18:44:56 [INFO] a
Sep 06 18:44:56 consul201-bison.node.consul consul[13244]: ==> Consul agent running!

»Setup Consul environment variables

Notice that since TLS encryption is enabled, you will now need to use the server certificates to complete all other tasks.

  1. To make it easier to use the CLI for the remainder of the setup, and to complete the ACL bootstrapping process, set the following environment variables for all your agents.

    export CONSUL_CACERT=/etc/consul.d/consul-agent-ca.pem
    export CONSUL_CLIENT_CERT=/etc/consul.d/<dc-name>-<server/ client>-consul-<cert-number>.pem
    export CONSUL_CLIENT_KEY=/etc/consul.d/<dc-name>-<server/   client>-consul-<cert-number>-key.pem
    

»Bootstrap the ACL system

The Secure Consul with Access Control Lists (ACLs) tutorial provides instructions on configuring and enabling ACLs on new agents. The below guidance assumes Consul has already been started without ACLs configured.

»Create the initial bootstrap token

  1. Working from one agent generate the Consul bootstrap token, which has unrestricted privileges.

    $ consul acl bootstrap
    

    This will return the Consul bootstrap token. You will need the SecretID for all subsequent Consul API requests (including CLI and UI). Ensure that you save the SecretID.

  2. Set CONSUL_MGMT_TOKEN env variable.

    $ export CONSUL_HTTP_TOKEN="<Token SecretID from previous step>"
    $ export CONSUL_MGMT_TOKEN="<Token SecretID from previous step>"
    
  3. Create a node policy file (node-policy.hcl) with write access for nodes related actions and read access for service related actions.

    agent_prefix "" {
      policy = "write"
    }
    node_prefix "" {
      policy = "write"
    }
    service_prefix "" {
      policy = "read"
    }
    session_prefix "" {
      policy = "read"
    }
    
  4. Generate the Consul node ACL policy with the newly created policy file.

    $ consul acl policy create \
      -token=${CONSUL_MGMT_TOKEN} \
      -name node-policy \
      -rules @node-policy.hcl
    
  5. Create the node token with the newly created policy.

    $ consul acl token create \
      -token=${CONSUL_MGMT_TOKEN} \
      -description "node token" \
      -policy-name node-policy
    
  6. On all Consul Servers add the node token.

    $ consul acl set-agent-token \
      -token="<Management Token SecretID>" \
      agent "<Node Token SecretID>"
    

»Apply Enterprise license

When using the Enterprise version of Consul binaries, there is a time limit Consul will function without applying a license. For Enterprise customers that have purchased a license, work with your Technical Account Manager (TAM) to get the appropriate license. To apply the license you'll need to follow the below steps.

  1. Ensure you are using an Enterprise binary.

    The license commands are only available in enterprise versions of the binaries. Make sure enterprise versions of the binaries have been installed and are running on the servers.

    consul version
    

    Consul Enterprise version will end with +ent.

    Consul v1.7.2+ent
    Protocol 2 spoken by default, understands 2 to 3 (agent will automatically use protocol >2 when speaking to compatible agents)
    
  2. Run the following command on the Consul leader.

    $ consul license put "<paste license text here>"
    

    Once the command runs successfully, the output should indicate that the license is valid.

    License is valid
    License ID: a763ad9f-ecfa-948c-7355-5a502aeac721
    Customer ID: 7e6ac3e0-f44b-23c3-c486-3264eabc4326
    Expires At: 2018-06-30 06:59:59.999 +0000 UTC
    Datacenter: *
    Package: cHJlbWl1bQ==
    Licensed Features:
        Automated Backups
        Automated Upgrades
        Enhanced Read Scalability
        Network Segments
        Redundancy Zone
        Advanced Network Federation
    

    The license will be replicated to other the Consul servers from the leader.

»Next steps

In this tutorial you configured servers and clients in accordance to the reference architecture. This is the first step in deploying your first datacenter. In the next tutorial, you will learn how to configure backups to ensure the datacenter state is protected against possible failures.

To increase security for your datacenter, we recommend completing the Secure Consul with Access Control Lists (ACLs), Secure Gossip Communication with Encryption, and Secure Consul Agent Communication with TLS Encryption. All three tutorials are in the Day 1 learning path.

Finally, we also recommend reviewing the Run Consul as a Service on Windows tutorial for a mixed workload environment.