Connect a Consul Client to HCP Consul | Consul

»Prerequisites

This tutorial assumes you have an HCP Consul cluster deployed in your HCP account, and that you have correctly configured an AWS peering connection between your HVN to your AWS VPC. The peering connection allows your HCP Consul servers deployed in your Consul cluster to communicate with clients that you deploy in your AWS VPC. Your clients must have a valid L3 route configured that allows them to communicate with your HCP Consul cluster's private IP. They cannot use a public IP even if you have one configured.

Review the manual deployment tutorial or the automatic deployment tutorial for guidance on creating the necessary network configuration. To complete this tutorial, we expect that you have already provisioned the following resources.

An HCP HashiCorp Virtual Network (HVN)
An HCP Consul cluster
An AWS VPC
An AWS peering connection between your HVC and your VPC
An AWS EC2 instance deployed to the peered VPC

If you do not yet have and EC2 instance provisioned in the target VPC, refer to the official AWS Documentation for guidance on how to create an EC2 instance that you can use with this tutorial.

Note: In this tutorial, you will add a Consul client running on an AWS virtual machine (EC2) instance. This instance must be deployed to the same VPC that you have peered with your HVN.

»Retrieve client configuration

You can use the HCP Portal to retrieve the client configuration information you need to connect your client agent to your HCP Consul cluster. Navigate to the Consul resource page in the HCP portal, and then select the Consul cluster you want to join your client to. Click the Download client config button to download a zip archive that contains the necessary files to join a client to the cluster. The archive includes a default client configuration and certificate. Both should be considered secrets, and should be kept in a secure location.

Note: This archive will be downloaded to whatever machine you used to access the HCP Portal. You will have to upload the archive to your EC2 instance for use there.

»Install Consul

If you have not done so already, you should start a terminal session on your EC2 instance. Refer to the official AWS documentation for details on how to connect to your EC2 instance.

Once you are connected to your EC2 instance, follow the official Consul installation instructions to install the Consul binary.

»Upload configuration to EC2 instance

You will need to copy the configuration file you downloaded from the HCP Portal to the EC2 instance. The following example is one way you could do that using scp with an ubuntu based EC2 instance. You may have to adapt the command if you chose a non-ubuntu machine image. Issue this command from the machine and directory where you downloaded the client configuration from the HCP Portal.

$ scp -i /path/to/your/aws.pem client_config_bundle_consul_<YOUR_CONSUL_CLUSTER_NAME>.zip ubuntu@<YOUR_EC2_INSTANCE_PUBLIC_IP>:/home/ubuntu

Switch back to the ssh session you initiated with your EC2 instance, and change directory to the directory where you uploaded the client configuration archive. Issue the following command to extract the configuration files from the archive you just uploaded.

$ unzip client_config_bundle_consul_<YOUR_CONSUL_CLUSTER_NAME>.zip

List the contents of the directory to make sure you have all the necessary files.

$ ls
ca.pem  client_config.json client_config_bundle_consul_<YOUR_CONSUL_CLUSTER_NAME>.zip

Now remove the zip archive as it contains sensitive information, and you will no longer need it. Do this on both the machine where you first downloaded the archive, and on the EC2 instance that you uploaded it to.

$ rm client_config_bundle_consul_<YOUR_CONSUL_CLUSTER_NAME>.zip

»Review configuration

The following section explains the variable security and datacenter settings contained within the client_config.json file. Here is an example of the configuration contained within that file.

{
  "acl": {
    "enabled": true,
    "down_policy": "async-cache",
    "default_policy": "deny"
  },
  "ca_file": "./ca.pem",
  "verify_outgoing": true,
  "datacenter": "dc1",
  "encrypt": "<GOSSIP_ENCRYPTION_KEY>",
  "server": false,
  "log_level": "INFO",
  "ui": true,
  "retry_join": [
    "<CONSUL_CLUSTER_PRIVATE_ENDPOINT>"
  ],
  "auto_encrypt": {
    "tls": true
  }
}

The acl.enabled setting is set to true which ensures that only requests with a valid token will be able to access resources in the datacenter. To add your client, you will need to configure an agent token. You will do that in a separate file later in the tutorial.
The auto_encrypt.tls setting is set to true to ensure transport layer security is enforced on all traffic with and between Consul agents.
The ca_file setting points to the ca.pem file you downloaded as part of the client configuration from the HCP Portal. You must make sure that the ca.pem that was part of the configuration bundle is available on any machine where you will run a client agent and exists at the path specified in this file. You will edit this setting later in the tutorial to point to the /consul/opt path.
The encrypt setting is set to your Consul cluster's gossip encryption key. Do not modify the encryption key that is provided for you in this file.
The retry_join is configured with the private endpoint address of your HCP Consul cluster's API. This is the address that your client will use to interact with the servers running in the HCP Consul cluster. Do not modify the value that is provided for you in this file.

»Create configuration directory

For this tutorial, you will use the /opt/consul directory to store all of your Consul configuration and data files. Issue the following command on your EC2 instance to create the folder.

$ sudo mkdir -p /opt/consul

Note: Since you will be storing all configuration files in this /opt/consul directory, you should edit the client_config.json file now, and make sure that the ca_file setting is set to /opt/consul/ca.pem.

»Create an ACL token

As mentioned previously, ACLs are enabled in your HCP Consul cluster because HCP is secure by default. In order for your datacenter client agent to join the HCP Consul cluster on start up, you must provide it an ACL token with the necessary privileges. Navigate to the Overview tab on your Consul cluster's details page in the HCP Portal, and then click the Generate token button. A new admin token will be generated and displayed in a modal dialog window. Copy the token, but be sure to store it somewhere safe, as this token will have broad privileges for your Consul data.

Note: If you are configuring a production environment, you should create a client token with a minimum set of privileges. For an in depth review of how to configure ACLs for Consul, refer to the Secure Consul with Access Control Lists tutorial or the official documentation.

»Create an ACL configuration file

Now that you have an ACL token, you can join your client to the HCP Consul cluster. Use the following json to generate a new configuration file on the EC2 instance named client_acl.json.

{
  "acl": {
    "tokens": {
      "agent": "<YOUR-ACL-TOKEN>"
    }
  }
}

Edit the client_acl.json file, and replace the <YOUR-ACL-TOKEN> placeholder text with the token you generated from the HCP Portal.

»Start the client

Your current working directory should now have all the necessary configuration files you need. List the files to make sure.

$ ls
ca.pem client_acl.json client_config.json

Review the following checklist to make sure you have all the required configuration artifacts, and that you understand what each of them provides.

The ca.pem file will be used to authenticate via mTLS.
The client_acl.json file has the ACL token that will be used to authorize with the ACL system.
The client_config.json file has the gossip encryption key, the path to where you are about to move the ca.pem file, and the retry_join endpoint that the agent needs to use to join the Consul cluster.

Issue the following command to move all the necessary Consul configuration files to the /opt/consul folder on your EC2 instance.

$ sudo mv ca.pem client_config.json client_acl.json /opt/consul

List the contents of the /opt/consul directory to verify the files were successfully moved.

$ ls /opt/consul
ca.pem client_config.json client_acl.json

You can now start the client agent using the consul agent command.

sudo consul agent -config-dir=/opt/consul -data-dir=/opt/consul

Note: For production environments, you will not use the consul agent command to start the agent. Refer to the Consul Deployment Guide for more information on how to configure Consul as a service.

Since ACLs are enabled, you will need a token when using the Consul CLI and API. Use the token you generated in the HCP portal for this tutorial.

You can use the Consul CLI to verify that your client agent on your EC2 instance was able to successfully register with your server agent in your HCP Consul cluster. You will need to open a new ssh session to your EC2 instance since the Consul process is now currently running in your original session. Since ACLs are enabled, you will need a token when using the CLI and API. You can use the token you saved in your /opt/consul/client_acl.json file.

$ cat /opt/consul/client_acl.json
{
  "acl": {
    "tokens": {
      "agent": "<YOUR-ACL-TOKEN>"
    }
  }
}

Note: An alternative to setting the CONSUL_HTTP_TOKEN environment variable is to use the -token flag after any Consul CLI command. See the Consul CLI reference for additional information.

Use the Consul CLI to verify that you have a Consul deployment with at least 1 server and 1 client.

Copy the ACL token, and pass it to the consul members command as the -token option.

$ consul members -token <YOUR-ACL-TOKEN>
Node              Address             Status  Type    Build      Protocol  DC   Segment
ip-172-25-20-128  172.25.20.128:8301  alive   server  1.9.3+ent  2         dc1  <all>
ip-10-2-160-249   10.2.160.249:8301   alive   client  1.9.3      2         dc1  <default>

Congratulations! You have successfully peered a Consul agent with your fully managed HCP Consul cluster.

»Next steps

In this tutorial, you deployed a Consul client and connected it with the servers running in your HCP Consul cluster. To learn more about Consul's features, and for step-by-step examples of how to perform common Consul tasks, complete one of the Get Started with Consul tutorials.

If you encounter any issues, please contact the HCP team at support.hashicorp.com.

Infrastructure

Security

Networking

Applications