You can use Vault's PKI Secrets Engine to generate and renew dynamic X.509 certificates. Aside from being useful to secure Consul RPC and Raft communication, the PKI secrets engine can also act as a Certificate Authority (CA) for Consul Connect service mesh.
One of the benefits of using a service mesh is to have certificates for the service instances generated and rotated automatically and used to authenticate and secure service communication. Using Vault as CA for your service mesh allows you to use unified workflow for certificate generation and have a unified control over all Consul certificates' lifecycle.
In this tutorial, you will learn how to configure your Consul datacenter to use Vault as the service mesh CA for new and existing data centers. This tutorial gives you two paths: Using self-managed Consul and Vault cluster. Or using Consul and Vault clusters deployed to HashiCorp cloud.
»Prerequisites
This tutorial is divided into sections: One section for readers using self-managed Vault and Consul clusters. The other section is for readers wanting, or already using Consul and Vault hosted on HashiCorp Cloud (HCP)
»Self Managed
»HashiCorp Cloud (HCP)
aws-cli +2.4.15
openssl +3.0.1
kubectl +v1.22.4
terraform +1.0.6
tar (GNU tar) +1.26
(Required forkubectl cp
)- If using Windows as your local environment's operating system, review this StackOverflow thread on using
kubectl cp
to/from Windows environments. git +v2.30.1
- A Consul Enterprise license. Request a trial license on the Consul Enterprise trial registration page..
- For AWS CLI, a credentials file located at
$HOME/.aws/credentials
file is needed to run the terraform project, and kubectl. - HCP Service Principal Credentials. Learn how to create Service Principals by reading the HCP Docs Service Principals documentation page.
In this tutorial you will configure the Consul Connect Certificate Authority in two setups: First, you will deploy HCP Consul and configure the HCP Consul Connect CA with HCP Vault after the cluster has deployed, changing the Certificate Authority after the HCP Consul cluster is running.
Then, you will deploy a new Consul Enterprise cluster to Amazon Elastic Kubernetes Service (EKS) as a client to the HCP Consul cluster. You will configure the same HCP Vault cluster as the Consul Connect CA for this Consul Cluster.
In this configuration, HCP Consul Connect uses HCP Vault's root certificate for identity verification. The client cluster on EKS will use this root certificate to register and identify with HCP Consul acting as the server. The HCP Consul instance signs an intermediate certificate for the client cluster on EKS. This intermediate certificate signs leaf certificates for services deployed to the service mesh on the client cluster. You will then deploy the HashiCups application to the Consul Connect service mesh on EKS.
Reference the following diagram for the services you will deploy in this tutorial.
A repository including terraform code for this tutorial is provided to complete your tasks. The terraform code deploys an HCP Consul Cluster, an HCP Vault Cluster, an Amazon EKS Cluster, and the underlying networking components for HCP and AWS to communicate with each other. These resources accrue charges in your HCP and AWS accounts. A cleanup script is provided at the end of this tutorial.
If you do not already have a HashiCorp Cloud account, sign up at the HashiCorp Cloud registration page. New accounts receive $50 in account credit, an amount sufficient for the HCP resources deployed during this tutorial.
NOTE: HCP Credits do not cover the associated cost of the AWS resources deployed in this tutorial.
To begin, clone the repository.
- HTTPS
- SSH
Navigate into the repository folder.
Fetch the tags from the git remote server, and checkout the tag for this tutorial.
Navigate into the project folder for this tutorial.
Place your Consul Enterprise license into the folder, from wherever you currently store it, and name it consul.hclic
.
Set your HCP service principal credentials as environment variables.
The terraform project consists of two deployment steps. The first terraform init and apply deploys the HCP infrastructure and Amazon EKS. The second terraform init and apply creates a pod inside Amazon EKS from data generated from the first terraform project, for you to use during the tutorial.
Initialize the first terraform project.
Run terraform apply to deploy the project, which presents a confirmation screen to deploy the bulk of the resources for this tutorial.
Type "yes", to deploy all the resources. Then, run terraform init and apply once again inside the inside the working-environment
directory to deploy Kubernetes specific resources for you to complete the tasks in this tutorial.
The deployment takes up to 40 minutes to complete. During this time, check out this Community Office Hours recording which discusses setting up Vault as a certificate authority for a Consul cluster.
»Your workbench
You will access HCP Vault and HCP Consul from inside the AWS EKS Cluster deployed with terraform. A pod is provided for you with the tools to complete the tasks in this tutorial. Going forward, this pod is referred to as your "workbench." When workbench is referenced, it specifically means the environment inside your working environment pod.
The workbench on the deployed EKS cluster provides three pre-installed text editors: nano, emacs, and vim. If you are
unfamiliar with these editors, or have a preference for another editor, the yum
package manager is available for you
to install packages on the container.
The workbench is available as soon as terraform apply
finishes.
Save the identifier of the workbench as a shell variable, by searching for the workbench's app
label.
Copy your Consul Enterprise license to the workbench.
Login to the workbench's bash
shell.
The workbench, deployed on the EKS cluster, is where you will complete most of the tasks for this tutorial.
The workbench has automatically configured an IAM Role for the preinstalled AWS CLI. You will be able to access all AWS EKS resources needed during this tutorial.
List the pods in the EKS cluster with kubectl
confirming EKS
resources are available inside the workbench.
The workbench automatically pre-loads the HashiCorp Cloud tokens necessary to communicate with the HashiCorp Cloud (HCP)
resources. Verify HCP Vault is available by using Vault's status
command.
Verify HCP Consul is available by using Consul's members
command.
»HCP Consul Connect Certificate Authority
When Consul Connect is initially enabled on HCP Consul, it self-signs its own TLS certificates. In this section, you are configuring HCP Consul to use HCP Vault as the Certificate Authority.
While an HCP Consul Cluster is active, configuration changes to the HCP Consul service mesh Certificate Authority are actively ignored. This includes when an HCP Consul instance is reloaded. Or if the underlying HCP hosts are restarted. Instead, Consul service mesh provides a specific API endpoint and Consul Connect CLI command to update the certificate authority while the Consul cluster is active.
To observe HCP Consul service mesh's current self-signing status, use the CLI or Consul REST API to return the CA configuration.
- Consul CLI
- Consul REST API
»Manage HCP Vault pki secrets engine paths
In this section you will begin to configure HCP Vault as the HCP Consul Connect certificate authority. You will create
an HCP Vault secrets engine for the root certificate in the /connect_root
pki path, and a leaf secrets engine, in the
/connect_inter
pki path, both in HCP Vault.
The root certificates is hosted in HCP Vault, and managed by HCP Consul. The root certificate is used to verify the identity of the clusters, and to sign intermediate certificates, also known as leaf certificates. Leaf certificates are used for services inside the Consul Connect service mesh.
In the following example, the Vault paths
represent the Consul service mesh root
authority as connect_root
. The intermediate authority path is represented
as connect_inter
. These values are customizable if you have alternative
requirements in non-tutorial environments.
Create the HCP Consul Connect policy file.
Add the following configuration to the policy file.
Implement the policy definition using the HCP Vault CLI.
»Create HCP Vault ACL token
After the policy is created, generate the HCP Vault policy token HCP Consul uses to access and manage the PKI secrets in HCP Vault. When using HCP Vault, you are required to specify the Vault namespace where this token is scoped. This tutorial uses the primary HCP Vault namespace admin.
Create the Consul ACL token with the Vault CLI, using the connect-ca
policy you created in the previous section, and saving it
as an environment variable.
Create the Consul Connect CA configuration file.
Output the environment variables to the terminal, validating they are not empty.
The root and Intermediate PKI path values need to match the paths set for the Vault ACL
token policy you created previously in this tutorial. Copy the configuration below into the
configuration file. Replace the environment variable placeholders with the corresponding shell output, as displayed
in the echo statement above. For example, $VAULT_NAMESPACE
is replaced with admin
.
1 2 3 4 5 6 7 8 9 10111213141516
Implement this HCP Consul configuration with the Consul Connect CLI.
»Verify the HCP Consul Connect Certificate Authority has changed
After implementing the configuration, the certificate authority immediately updates. To confirm and observe, use the Consul Connect CLI, or REST API to query the configuration for Consul Connect CA.
- Consul CLI
- Consul HTTP REST API
»Configure HCP Vault to HCP Consul Connection
To connect to HCP Vault in Kubernetes, the Kubernetes cluster needs a Vault agent installed on the cluster to pass the stored certificates to the Consul client cluster.
»Install Vault agent on EKS cluster
Using the vault-k8s
helm chart, deploy a Vault agent to the cluster.
Before deploying Consul onto Kubernetes and connecting to HCP Consul, a Vault agent is installed on your EKS cluster, serving as the link between HCP Vault and the Consul client cluster. The agent retrieves certificates for the Consul client cluster when requested.
Create a values file for the Vault agent, and output your HCP Vault's URL.
Replace $VAULT_ADDR
with the contents of the $VAULT_ADDR
environment variable in your workbench.
In this tutorial, it is the private URL for the HCP Vault cluster. Copy the following configuration into the values
file.
12345678
Add the Hashicorp repository to helm, to install the Vault agent on EKS.
Deploy the Vault agent to Kubernetes.
After deployment, the Vault agent searches for the $VAULT_TOKEN
environment variable that is set in your environment,
to create the initial connection to HCP Vault. This is an administrator token. In production environments, consider
using an alternative token with limited permissions.
NOTE: The $VAULT_TOKEN
has a default ttl
policy of six hours. If you do
not complete this tutorial within six hours, the HCP Vault agents will be unable
to connect to HCP Vault, and you will need to generate a new token.
Validate the vault-agent-injector
pod is in a working state with kubectl.
After the Vault agent is installed, a service account is created in the EKS cluster. You will use information from this service account in the next section. Validate the service account was created by using the following command.
»Configure Kubernetes auth engine for HCP Vault
You will now configure the Kubernetes auth engine on HCP Vault with the roles and policies required for the Vault agent to connect to HCP Vault.
For the EKS Client Cluster to retrieve secrets and TLS certificates from HCP
Vault, enable the Kubernetes auth engine, and the kv
secrets engine using the Vault CLI.
»Create HCP Vault roles and policies
Next, create a role and policy in HCP Vault that maps to the Consul service account in Kubernetes. The Consul service account does not exist yet, so you calculate the name of the service account. Next, you will save several environment variables that are needed to configure the role and policy.
Retrieve data stored in the Vault service account on Kubernetes to update the Kubernetes auth engine in HCP Vault. This adds the service account's certificate and token to the auth engine in HCP Vault, and configures the auth engine with the Kubernetes control plane URL.
Set the following environment variables in your shell environment. Update the Kube auth engine in HCP Vault using these values.
The JSON Web Token is used to authenticate the service account to HCP Vault.
The JWT's CA Certificate handles identity verification between HCP Consul and HCP Vault, for the Vault agent.
The control plane URL is used by the Vault agent to notify HCP Vault the location of the Kubernetes cluster.
Update the Kubernetes Auth Engine in HCP Vault with these environment variables.
»Configure HCP Vault role
Begin configuring the HCP Vault role.
You will pre-render a helm template for Consul on Kubernetes to calculate the Consul service account name, using this name to create an HCP Vault role and policy.
Create the name of the Consul release and the name of the HCP Vault role. Set this name as an environment variable.
Pass the release name variable to the helm template command, specifying the client-daemonset template, which creates the Consul service account for Kubernetes.
This command doesn't create the service account, the service account is created when you install Consul on Kubernetes later in this tutorial.
Store the JWT's CA Certificate in Vault KV.
Validate the existence of the certificate.
Consul service mesh uses gossip encryption to communicate with HCP Vault in order to securely transport TLS leaf certificates. The gossip encryption key is generated by HCP Consul while your terraform code was deploying, and is pre-loaded into your environment shell.
Retrieve the value from the environment.
Store the secret in vault kv
Validate the secret is written to HCP Vault.
Create an HCP Vault policy that gives the Consul service account read access to the gossip encryption key stored in
Vault kv, and the CA certificate of the service account JWT. Attach the policy to a new role with the same name as the consul service
account. This role uses the connect-ca
policy you created earlier in this tutorial.
Create the policy file.
Add the following policy configuration to the policy file.
Write this policy to HCP Vault.
Attach the policy to a new role with the same name as the consul service
account. This role uses the connect-ca
policy you created earlier in this tutorial.
»Deploy Consul Client Cluster to Amazon EKS
HCP Consul service mesh CA is using HCP Vault. You will next configure a Consul client cluster that registers to HCP Consul. This client cluster issues leaf certificates signed by the root certificate and assigns them to services inside the Consul service mesh.
If you deploy the Consul client cluster with Consul service mesh enabled, but do not configure the Consul Connect CA provider, installation will fail. Client clusters that register with Consul Connect server cluster in the same datacenter need to present the root certificate managed by HCP Consul. If you do not configure your client cluster, it will self-sign its own certificates, and present that certificate to HCP Consul, which fails. The role and policies you created in HCP Vault permit the Consul client cluster to retrieve and present the root certificate to HCP Consul.
Before installing the Consul client cluster, prepare the Kubernetes cluster with secrets generated by HCP Consul. The Consul client cluster accesses these values during installation when registering to the HCP Consul cluster. These values are preloaded into your shell as environment variables.
Seed the Kubernetes cluster with the HCP Consul CA certificate, the HCP Consul bootstrap token, the gossip encryption key, and your Consul Enterprise license. These items are stored as Kubernetes secrets.
Store the Consul CA certificate. (Note: This is separate from the Consul Connect CA Certificate)
Store the HCP Consul bootstrap token.
Store the Consul gossip encryption key
Store the Consul enterprise license
Create a Helm chart file for installing the Consul client cluster.
The helm chart for Consul on Kubernetes includes a global.secretsBackend
stanza to specify HCP Vault as the Consul
client's Connect CA backend. You will provide HCP Vault's information including the Root and Intermediate PKI paths
created in HCP Vault, with the roles created in HCP Vault for Consul, you created in an earlier section.
Edit the file with the following configuration, replacing $CONSUL_HTTP_ADDR
, $KUBE_CONTROL_PLANE
, $VAULT_ADDR
with their
outputs. Remove https://
from the output of $CONSUL_HTTP_ADDR
. Example output is shown below.
The above command prints similar output to the screen as shown in example output below.
1 2 3 4 5 6 7 8 9 1011121314151617181920212223242526272829303132333435363738394041424344454647484950515253545556575859606162636465666768
The workbench includes the Consul K8s CLI. The Consul K8s CLI is an alternative to installing Consul on Kubernetes instead of using Helm, Kubectl, and the Consul CLI.
Deploy Consul with the Consul K8s CLI. Press 'y' to accept and proceed with the installation.
»Verify Client Cluster Consul Connect Root Certificate
When the installation finishes, the HCP Consul Cluster has accepted the Amazon EKS Cluster presenting the Connect CA root certificate from HCP Vault for identity verification. To observe the Connect CA provider origin on the Consul client cluster, use the following command to query the Consul CLI.
Use the Consul Connect CLI to output the Consul Connect CA provider for the client cluster. Use the
consul connect ca get-config
command on any of the consul pods deployed to EKS.
»Deploy HashiCups with leaf certificates
With the root Connect CA now in place for the Consul client cluster, deploy HashiCups to the cluster. After, you will verify the generated leaf certificates, and the signing authority of those leaf certificates.
The HashiCups plan files are available in your workbench, located in the /tmp/hashicups/
directory.
Since ACLs are enabled, first apply Consul service intentions for the HashiCups services to have the ability to communicate with each other.
Deploy the HashiCups services to Kubernetes.
Confirm the HashiCups services are registered with Consul.
Confirm the HashiCups pods are in a working state, with kubectl. This can take several moments as the services scale and transfer into a "running" state.
»Validate leaf certificates for HashiCups services
With the HashiCups services deployed, verify that the leaf certificates are generated for the HashiCups services. You will use
the postgres service to confirm. Use the /v1/agent/connect/ca/leaf
Consul HTTP API endpoint to retrieve the leaf
certificate bundle for this service inside the service mesh. Append the name of the service to the end of the
API path. In this setup, use the /v1/agent/connect/ca/leaf/postgres
endpoint to retrieve the certificate bundle.
NOTE: To learn more about certificate chains, bundles and their structure, read the Wikipedia article on the X.509 standard.
Observe the output of the saved file. (Edited to save space)
This bundle includes the certificate signing authority for the service, the intermediate CA certificate in the bundle, and
certficate for the authority which signed the intermediate certificate. This bundle ensures the sender and Certificate
Authority's trustworthiness. The first, or top certificate in the chain represents the postgres
service in the chain.
Copy the top, or first certificate into its own file, starting from the BEGIN CERTIFICATE
line at the top, to the
first END CERTIFICATE
line.
Inspect the certificate with the openssl
tool, specifically observing the Signature Algorithm
and X509v3 extensions
sections of the command output.
By referencing the Standard Extensions in the X509 extensions configuration guide,
you can confirm that this service certificate is not a certificate assigned to a Certificate Authority, via the absence of the
"Basic Extensions" section in this output. This means the certificate was issued and signed by a CA. Verify the origin
of this certificate by observing the value of X509v3 Authority Key Identifier
and save it as a shell variable.
This serial number represents the X509v3 Subject Key Identifier
for the next certificate in the bundle. This serial number
is not available to query in HCP Vault directly. You will retrieve this serial number indirectly. First, return to the
certificate bundle you saved at the beginning, and copy the contents of the two remaining certificates into
separate files. Inspect each file's X509v3 Subject Key Identifier
value.
The middle certificate's Subject Key Identifier
does not match the service certificate's Authority Key Identifier
.
Inspect the certificate at the bottom of the certificate bundle.
WARNING Depending on the order returned in the certificate bundle, the middle certificate may match the authority key identifier. If so, skip to the next step for the last certificate in the bundle to complete the task.
This certificate's Subject Key Identifier
matches the Authority Key Identifier
for the postgres
service.
Query the connect_inter
pki path for this certificate's Serial Number
value, replacing the colons with slashes. The
replacement allows the serial number to be included in as a search parameter for the Vault HTTP API.
Query the value of the Serial Number in the HCP Vault API.
This shell output confirms the certificate generated for the postgres
service is signed by an authority that has its CA
certificate stored in HCP Vault. In this setup, the signing authority is the Consul Connect service mesh Certificate
Authority backed by HCP Vault, as previously configured.
As a challenge, trace the intermediate certificate to the root certificate by using these directions on the middle certificate listed in the tutorial, and final certificate in the certificate chain, comparing the values against the middle certificate in the chain.
»Validate HashiCups deployment status.
Exit the workbench. You will verify the HashiCups services are functioning as expected.
Update your workbench's kubeconfig
file with your cluster information.
Verify the rest of the HashiCups pods are in a working state with kubectl.
Use kubectl
to create a port forward for the frontend
, and public-api
pods to your local environment (Not the workbench). This example
sets a shell variable for the name of the service, passing it to kubectl's port-forward
command. This creates a
connection from the local pod port, to your local environment.
Navigate to the frontend URL http://localhost:3500.
You can now observe the HashiCups application deployed to the client cluster, in your local browser.
»Leaf certificate expiration and verification
The /agent/connect/ca/leaf/$SERVICE
REST API endpoint is used to return information about the leaf certificate
generated for the Consul service.
Log back in to your workbench.
To set a leaf certificate's expiration, the leaf_cert_ttl
option is set in the helm chart values file. This
configuration is available to set in the free-text secretsBackend.vault.connectCA.additionalConfig
stanza in
consul-values-client.yaml
. This setting applies to all leaf certificates signed by the Consul Connect CA. The minimum
rotation period for leaf certificates is one hour. This process is managed automatically by HCP Consul and HCP Vault.
Verify a certificate's TTL by inspecting the response when calling the /agent/connect/ca/leaf/$SERVICE
endpoint, with
curl.
The time difference between ValidAfter
and ValidBefore
should match the value of the configured TTL. In this case,
the value of these keys is 72 hours and 30 seconds, matching the configured value of 72 hours, with 30 seconds set
as a buffer by the CA.
»Cleanup:
Exit the workbench.
Run terraform destroy in both projects to tear down the environment
This can take approximately 15 minutes to complete.
»Next Steps
In this tutorial you configured Consul Connect service mesh's certification Authority using the PKI secrets engine in Vault. You configured a root certificate for Consul Connect backed by Vault, and generated leaf certificates for services deployed to Consul, signed by this root certificate.
Continue learning about the options of securing Consul by taking additional tutorials on Learn. The Secure Consul Gossip Communication with Vault tutorial teaches you to secure gossip protocol communication inside your Consul clusters. The Generate mTLS Certificates for Consul using Vault tutorial helps you implement mutual TLS for the overall Consul cluster.
Learn more about ACL token administration on Vault with the Generate Consul Tokens with HashiCorp Vault tutorial.