Vault as Consul Service Mesh Certification Authority

You can use Vault's PKI Secrets Engine to generate and renew dynamic TLS X.509 certificates. Aside from being useful to secure Consul's RPC and Raft communication, the PKI secrets engine provides support serving as a Certificate Authority for Consul service mesh.

One of the benefits of using a service mesh is creating, generating, and rotating TLS certificates for services in the service mesh. Using Vault as the CA for your service mesh provides unified management for TLS certificates.

This tutorial configures a Consul datacenter to use Vault as the Consul service mesh CA for new and existing Consul datacenters.

»Prerequisites

To test the commands provided by this tutorial you need a Consul datacenter configured with ACL, TLS encryption, and Gossip encryption enabled. You can use the following configuration file as an example to set up a single node Consul datacenter.

# Datacenter configuration
domain = "consul"
datacenter = "dc1"
node_name = "consul-server-0"

data_dir = "/<absolute-path-to>/consul/data"

# Server configuration
server = true
bootstrap_expect = 1

ui_config {
  enabled = true
}

ports {
  grpc = 8502
}

# TLS Encryption configuration
verify_incoming        = false
verify_incoming_rpc    = true
verify_outgoing        = true
verify_server_hostname = true

ca_file   = "/<absolute-path-to>/consul-agent-ca.pem"
cert_file = "/<absolute-path-to>/dc1-server-consul-0.pem"
key_file  = "/<absolute-path-to>/dc1-server-consul-0-key.pem"

auto_encrypt = {
    allow_tls = true
}

# ACL configuration
# After startup, bootstrap the ACL system with `consul acl bootstrap` command
acl = {
  enabled = true
  default_policy = "deny"
  enable_token_persistence = true
}

# Gossip Encryption
encrypt = "tRgXSb6ClvzV1myhc75rWIdwx8tTmUI8UxySKecxiQA="

# If running Consul 1.8.x or below, enable central service configuration
enable_central_service_config = true

# You can Append the Connect configuration here or place it in a separate
# configuration file.
# ...

consul-config.hcl

# Datacenter configuration
domain = "consul"
datacenter = "dc1"
node_name = "consul-server-0"

data_dir = "/<absolute-path-to>/consul/data"

# Server configuration
server = true
bootstrap_expect = 1

ui_config {
  enabled = true
}

ports {
  grpc = 8502
}

# TLS Encryption configuration
verify_incoming        = false
verify_incoming_rpc    = true
verify_outgoing        = true
verify_server_hostname = true

ca_file   = "/<absolute-path-to>/consul-agent-ca.pem"
cert_file = "/<absolute-path-to>/dc1-server-consul-0.pem"
key_file  = "/<absolute-path-to>/dc1-server-consul-0-key.pem"

auto_encrypt = {
    allow_tls = true
}

# ACL configuration
# After startup, bootstrap the ACL system with `consul acl bootstrap` command
acl = {
  enabled = true
  default_policy = "deny"
  enable_token_persistence = true
}

# Gossip Encryption
encrypt = "tRgXSb6ClvzV1myhc75rWIdwx8tTmUI8UxySKecxiQA="

# If running Consul 1.8.x or below, enable central service configuration
enable_central_service_config = true

# You can Append the Connect configuration here or place it in a separate
# configuration file.
# ...

More details on how to securely configure your consul datacenter are available in the following tutorials:

Vault: the tutorial assumes you already have a running Vault 0.10.3+ cluster in your network. You can use a local Vault dev agent or an existing Vault deployment.

Launch Terminal

This tutorial includes a free interactive command-line lab that lets you follow along on actual cloud infrastructure.

Security Warning: This tutorial is not for production use. By default, the lab will install an insecure configuration of Consul.

»Configure Vault ACL

To interact with the PKI Secrets Engine endpoints, you have to generate a Vault token, giving the appropriate permissions to Consul to generate the certificates.

»Create Vault policies

The following Vault policy allows Consul to create and manage the PKI paths in Vault. Consul is granted the ability to create the PKI mount points and given full control of the root and intermediate CA and to generate leaf certificates for Consul service mesh clients.

The leaf certificates are created for each service instance and are used to verify the identity of the services and to secure communication across service instances.

In this example the RootPKIPath is connect_root and the IntermediatePKIPath is connect_inter. These values should be updated for your environment.

# Consul Managed PKI Mounts
path "/sys/mounts" {
  capabilities = [ "read" ]
}

path "/sys/mounts/connect_root" {
  capabilities = [ "create", "read", "update", "delete", "list" ]
}

path "/sys/mounts/connect_inter" {
  capabilities = [ "create", "read", "update", "delete", "list" ]
}

path "/connect_root/*" {
  capabilities = [ "create", "read", "update", "delete", "list" ]
}

path "/connect_inter/*" {
  capabilities = [ "create", "read", "update", "delete", "list" ]
}

vault-policy-connect-ca.hcl

# Consul Managed PKI Mounts
path "/sys/mounts" {
  capabilities = [ "read" ]
}

path "/sys/mounts/connect_root" {
  capabilities = [ "create", "read", "update", "delete", "list" ]
}

path "/sys/mounts/connect_inter" {
  capabilities = [ "create", "read", "update", "delete", "list" ]
}

path "/connect_root/*" {
  capabilities = [ "create", "read", "update", "delete", "list" ]
}

path "/connect_inter/*" {
  capabilities = [ "create", "read", "update", "delete", "list" ]
}

Save the configuration in a file named vault-policy-connect-ca.hcl and create the policy with the Vault CLI.

$ vault policy write connect-ca vault-policy-connect-ca.hcl

Success! Uploaded policy: connect-ca

$ vault policy write connect-ca vault-policy-connect-ca.hcl

Success! Uploaded policy: connect-ca

»Create Vault token

Once the policy is created, create a Vault token to use for the integration.

$ vault token create -policy=connect-ca

Key                  Value
---                  -----
token                s.hqXGLH9OhQFKVb7avZtlDHzq
token_accessor       DwK2Zvj8kHKpLd8w6xiI2AWM
token_duration       768h
token_renewable      true
token_policies       ["connect-ca" "default"]
identity_policies    []
policies             ["connect-ca" "default"]

$ vault token create -policy=connect-ca

Key                  Value
---                  -----
token                s.hqXGLH9OhQFKVb7avZtlDHzq
token_accessor       DwK2Zvj8kHKpLd8w6xiI2AWM
token_duration       768h
token_renewable      true
token_policies       ["connect-ca" "default"]
identity_policies    []
policies             ["connect-ca" "default"]

»Setup your environment

If you use a remote deployment, you can configure a local Consul binary to interact with the datacenter. Set the CONSUL_HTTP_ADDR variable on your local machine or jump host to the IP address of a server.

$ export CONSUL_HTTP_ADDR=<consul_server_ip>:<consul_server_port>

$ export CONSUL_HTTP_ADDR=<consul_server_ip>:<consul_server_port>

The same applies to the ACL token to access Consul data. You can export the token with the CONSUL_HTTP_TOKEN variable.

$ export CONSUL_HTTP_TOKEN=<consul_token>

$ export CONSUL_HTTP_TOKEN=<consul_token>

»Configure Consul service mesh CA

Once Vault is configured and the necessary policies and tokens are created, configure Consul to use Vault PKI secrets engine as Consul service mesh CA.

The suggested approach for using Vault as Consul service mesh CA is to configure it directly when deploying a new Consul datacenter.

»Configure Consul servers

Configure the service mesh CA using the connect stanza in the Consul configuration file.

## Service mesh CA configuration
connect {
  enabled = true
  ca_provider = "vault"
    ca_config {
        address = "<Your-Vault-address>"
        token = "<Your-Vault-token>"
        root_pki_path = "connect_root"
        intermediate_pki_path = "connect_inter"
        leaf_cert_ttl = "72h"
        rotation_period = "2160h"
        intermediate_cert_ttl = "8760h"
        private_key_type = "rsa"
        private_key_bits = 2048
    }
}

## Service mesh CA configuration
connect {
  enabled = true
  ca_provider = "vault"
    ca_config {
        address = "<Your-Vault-address>"
        token = "<Your-Vault-token>"
        root_pki_path = "connect_root"
        intermediate_pki_path = "connect_inter"
        leaf_cert_ttl = "72h"
        rotation_period = "2160h"
        intermediate_cert_ttl = "8760h"
        private_key_type = "rsa"
        private_key_bits = 2048
    }
}

Note: The leaf_cert_ttl defines the TTL for the certificates issued by the CA for Consul service mesh. The default value is 72h but using the parameter you can tune the lease duration to certificates. The shortest TTL value is 1h.

»Start Consul servers

After the configuration you can start the Consul servers making sure to include the connect configuration.

»Verify Consul service mesh CA configuration

Verify the CA settings by querying Consul for the CA configuration using either the Consul CLI or the REST API.

$ consul connect ca get-config

$ consul connect ca get-config

{
    "Provider": "vault",
    "Config": {
        "Address": "http://172.19.0.2:8200",
        "IntermediateCertTTL": "8760h",
        "IntermediatePKIPath": "connect_inter",
        "LeafCertTTL": "72h",
        "RootPKIPath": "connect_root",
        "RotationPeriod": "2160h",
        "Token": "password"
    },
    "State": null,
    "CreateIndex": 7,
    "ModifyIndex": 7
}

{
    "Provider": "vault",
    "Config": {
        "Address": "http://172.19.0.2:8200",
        "IntermediateCertTTL": "8760h",
        "IntermediatePKIPath": "connect_inter",
        "LeafCertTTL": "72h",
        "RootPKIPath": "connect_root",
        "RotationPeriod": "2160h",
        "Token": "password"
    },
    "State": null,
    "CreateIndex": 7,
    "ModifyIndex": 7
}

»Configure Consul clients

Once you configured Consul service mesh for the servers, you can configure the clients.

The configuration for the clients does not require to specify the provider but only to enable the service mesh feature.

connect {
  enabled = true
}

connect {
  enabled = true
}

Note: The client configuration is not tied to the CA provider in use by your Consul datacenter. This means that if you are changing the CA for an existing Consul datacenter, you will not need to apply any change to the already running clients.

»Verify certificates

Once Vault is configured to act as CA for Consul service mesh you can verify the certificates being used in your service mesh.

»Check root and intermediate certificates

Use the /agent/connect/ca/roots endpoint to retrieve the list of currently trusted root certificates.

$ curl -s \
  --header "X-Consul-Token: ${CONSUL_HTTP_TOKEN}" \
${CONSUL_HTTP_ADDR}/v1/agent/connect/ca/roots

$ curl -s \
  --header "X-Consul-Token: ${CONSUL_HTTP_TOKEN}" \
${CONSUL_HTTP_ADDR}/v1/agent/connect/ca/roots

{
  "ActiveRootID": "ee:93:e1:81:1f:bb:d2:ac:38:81:41:99:f8:be:f4:42:19:34:29:c2",
  "TrustDomain": "ec52b326-9aa4-5d2f-2032-c77aac1499b5.consul",
  "Roots": [
    {
      "ID": "d6:65:53:7c:16:df:0c:61:33:da:22:b2:55:01:34:3d:92:2d:cf:21",
      "Name": "Consul CA Root Cert",
      "SerialNumber": 9,
      "SigningKeyID": "bb:70:0e:1a:d3:33:3e:f0:d0:0b:36:22:dc:32:2f:1b:0d:41:24:47:fc:4d:d6:43:91:da:aa:19:ce:b9:b9:f3",
      "ExternalTrustDomain": "ec52b326-9aa4-5d2f-2032-c77aac1499b5",
      "NotBefore": "2021-07-22T13:39:00Z",
      "NotAfter": "2031-07-22T13:39:00Z",
      "RootCert": "-----BEGIN CERTIFICATE----- ... -----END CERTIFICATE-----\n",
      "IntermediateCerts": null,
      "Active": false,
      "PrivateKeyType": "ec",
      "PrivateKeyBits": 256,
      "CreateIndex": 11,
      "ModifyIndex": 54
    },
    {
      "ID": "ee:93:e1:81:1f:bb:d2:ac:38:81:41:99:f8:be:f4:42:19:34:29:c2",
      "Name": "Vault CA Root Cert",
      "SerialNumber": 17018489092300240000,
      "SigningKeyID": "60:db:cb:99:95:0d:39:b7:66:66:12:fa:6b:3b:24:39:9b:4f:ae:ad",
      "ExternalTrustDomain": "ec52b326-9aa4-5d2f-2032-c77aac1499b5",
      "NotBefore": "2021-07-22T13:38:59Z",
      "NotAfter": "2021-08-23T13:39:29Z",
      "RootCert": "-----BEGIN CERTIFICATE----- ... -----END CERTIFICATE-----",
      "IntermediateCerts": [
        "-----BEGIN CERTIFICATE----- ... -----END CERTIFICATE-----"
      ],
      "Active": true,
      "PrivateKeyType": "ec",
      "PrivateKeyBits": 256,
      "CreateIndex": 54,
      "ModifyIndex": 54
    }
  ]
}

{
  "ActiveRootID": "ee:93:e1:81:1f:bb:d2:ac:38:81:41:99:f8:be:f4:42:19:34:29:c2",
  "TrustDomain": "ec52b326-9aa4-5d2f-2032-c77aac1499b5.consul",
  "Roots": [
    {
      "ID": "d6:65:53:7c:16:df:0c:61:33:da:22:b2:55:01:34:3d:92:2d:cf:21",
      "Name": "Consul CA Root Cert",
      "SerialNumber": 9,
      "SigningKeyID": "bb:70:0e:1a:d3:33:3e:f0:d0:0b:36:22:dc:32:2f:1b:0d:41:24:47:fc:4d:d6:43:91:da:aa:19:ce:b9:b9:f3",
      "ExternalTrustDomain": "ec52b326-9aa4-5d2f-2032-c77aac1499b5",
      "NotBefore": "2021-07-22T13:39:00Z",
      "NotAfter": "2031-07-22T13:39:00Z",
      "RootCert": "-----BEGIN CERTIFICATE----- ... -----END CERTIFICATE-----\n",
      "IntermediateCerts": null,
      "Active": false,
      "PrivateKeyType": "ec",
      "PrivateKeyBits": 256,
      "CreateIndex": 11,
      "ModifyIndex": 54
    },
    {
      "ID": "ee:93:e1:81:1f:bb:d2:ac:38:81:41:99:f8:be:f4:42:19:34:29:c2",
      "Name": "Vault CA Root Cert",
      "SerialNumber": 17018489092300240000,
      "SigningKeyID": "60:db:cb:99:95:0d:39:b7:66:66:12:fa:6b:3b:24:39:9b:4f:ae:ad",
      "ExternalTrustDomain": "ec52b326-9aa4-5d2f-2032-c77aac1499b5",
      "NotBefore": "2021-07-22T13:38:59Z",
      "NotAfter": "2021-08-23T13:39:29Z",
      "RootCert": "-----BEGIN CERTIFICATE----- ... -----END CERTIFICATE-----",
      "IntermediateCerts": [
        "-----BEGIN CERTIFICATE----- ... -----END CERTIFICATE-----"
      ],
      "Active": true,
      "PrivateKeyType": "ec",
      "PrivateKeyBits": 256,
      "CreateIndex": 54,
      "ModifyIndex": 54
    }
  ]
}

When a datacenter is initialized, this will only list one trusted root. As in the example above, multiple roots may appear as part of certificate rotation or if the CA got migrated from Consul to Vault.

»Check leaf certificates for services

Use the /agent/connect/ca/leaf/<service> endpoint to retrieve the leaf certificates for the Consul service mesh services.

For example, to retrieve the certificate for the service web you can use the following.

$ curl -s \
    --header "X-Consul-Token: ${CONSUL_HTTP_TOKEN}" \
    ${CONSUL_HTTP_ADDR}/v1/agent/connect/ca/leaf/web

$ curl -s \
    --header "X-Consul-Token: ${CONSUL_HTTP_TOKEN}" \
    ${CONSUL_HTTP_ADDR}/v1/agent/connect/ca/leaf/web

{
  "SerialNumber": "59:3e:c6:7f:5c:2c:ac:d8:bb:50:d3:87:ec:3b:41:44:1d:9a:86:82",
  "CertPEM": "-----BEGIN CERTIFICATE----- ... -----END CERTIFICATE-----",
  "PrivateKeyPEM": "-----BEGIN EC PRIVATE KEY----- ... -----END EC PRIVATE KEY-----\n",
  "Service": "web",
  "ServiceURI": "spiffe://ec52b326-9aa4-5d2f-2032-c77aac1499b5.consul/ns/default/dc/dc1/svc/web",
  "ValidAfter": "2021-07-25T13:39:07Z",
  "ValidBefore": "2021-07-25T14:39:37Z",
  "CreateIndex": 486,
  "ModifyIndex": 486
}

{
  "SerialNumber": "59:3e:c6:7f:5c:2c:ac:d8:bb:50:d3:87:ec:3b:41:44:1d:9a:86:82",
  "CertPEM": "-----BEGIN CERTIFICATE----- ... -----END CERTIFICATE-----",
  "PrivateKeyPEM": "-----BEGIN EC PRIVATE KEY----- ... -----END EC PRIVATE KEY-----\n",
  "Service": "web",
  "ServiceURI": "spiffe://ec52b326-9aa4-5d2f-2032-c77aac1499b5.consul/ns/default/dc/dc1/svc/web",
  "ValidAfter": "2021-07-25T13:39:07Z",
  "ValidBefore": "2021-07-25T14:39:37Z",
  "CreateIndex": 486,
  "ModifyIndex": 486
}

»Certificate rotation and API caching

The /agent/connect/ca/leaf/<service> endpoint has a caching mechanism that might return stale results for service certificates. You can verify the result's age by inspecting the header data using the -verbose option with the curl command.

$ curl -s -verbose \
    --header "X-Consul-Token: ${CONSUL_HTTP_TOKEN}" \
    ${CONSUL_HTTP_ADDR}/v1/agent/connect/ca/leaf/web

$ curl -s -verbose \
    --header "X-Consul-Token: ${CONSUL_HTTP_TOKEN}" \
    ${CONSUL_HTTP_ADDR}/v1/agent/connect/ca/leaf/web

...
< HTTP/2 200
< age: 398999
< content-type: application/json
< vary: Accept-Encoding
< x-cache: HIT
< x-consul-default-acl-policy: deny
< x-consul-index: 1075
...

{
  "SerialNumber": "0e",
  "CertPEM": "-----BEGIN CERTIFICATE----- ... -----END CERTIFICATE-----",
  "PrivateKeyPEM": "-----BEGIN EC PRIVATE KEY----- ... -----END EC PRIVATE KEY-----\n",
  "Service": "web",
  "ServiceURI": "spiffe://7d131ccc-0605-10a2-cca2-c3b4eb89500f.consul/ns/default/dc/dc1/svc/web",
  "ValidAfter": "2021-07-22T13:39:07Z",
  "ValidBefore": "2021-07-22T14:39:37Z",
  "CreateIndex": 58,
  "ModifyIndex": 58
}

...
< HTTP/2 200
< age: 398999
< content-type: application/json
< vary: Accept-Encoding
< x-cache: HIT
< x-consul-default-acl-policy: deny
< x-consul-index: 1075
...

{
  "SerialNumber": "0e",
  "CertPEM": "-----BEGIN CERTIFICATE----- ... -----END CERTIFICATE-----",
  "PrivateKeyPEM": "-----BEGIN EC PRIVATE KEY----- ... -----END EC PRIVATE KEY-----\n",
  "Service": "web",
  "ServiceURI": "spiffe://7d131ccc-0605-10a2-cca2-c3b4eb89500f.consul/ns/default/dc/dc1/svc/web",
  "ValidAfter": "2021-07-22T13:39:07Z",
  "ValidBefore": "2021-07-22T14:39:37Z",
  "CreateIndex": 58,
  "ModifyIndex": 58
}

You can verify that the result is served by the cache using the x-cache header that, in this example shows a value of HIT.

In case you notice that the age header is bigger than the value defined for leaf_cert_ttl you can invalidate the cache by pointing at a different Consul node.

Change the CONSUL_HTTP_ADDR variable on your local machine or jump host to the IP address of a different server.

$ export CONSUL_HTTP_ADDR=<consul_server_ip>:<consul_server_port>

$ export CONSUL_HTTP_ADDR=<consul_server_ip>:<consul_server_port>

Re-issue the curl command to query the API endpoint. Leave the -verbose parameter set so to verify the cache headers.

$ curl -s -verbose \
    --header "X-Consul-Token: ${CONSUL_HTTP_TOKEN}" \
    ${CONSUL_HTTP_ADDR}/v1/agent/connect/ca/leaf/web

$ curl -s -verbose \
    --header "X-Consul-Token: ${CONSUL_HTTP_TOKEN}" \
    ${CONSUL_HTTP_ADDR}/v1/agent/connect/ca/leaf/web

...
< HTTP/2 200
< content-type: application/json
< vary: Accept-Encoding
< x-cache: MISS
< x-consul-default-acl-policy: deny
< x-consul-index: 486
...

{
  "SerialNumber": "59:3e:c6:7f:5c:2c:ac:d8:bb:50:d3:87:ec:3b:41:44:1d:9a:86:82",
  "CertPEM": "-----BEGIN CERTIFICATE----- ... -----END CERTIFICATE-----",
  "PrivateKeyPEM": "-----BEGIN EC PRIVATE KEY----- ... -----END EC PRIVATE KEY-----\n",
  "Service": "web",
  "ServiceURI": "spiffe://ec52b326-9aa4-5d2f-2032-c77aac1499b5.consul/ns/default/dc/dc1/svc/web",
  "ValidAfter": "2021-07-25T13:39:07Z",
  "ValidBefore": "2021-07-25T14:39:37Z",
  "CreateIndex": 486,
  "ModifyIndex": 486
}

...
< HTTP/2 200
< content-type: application/json
< vary: Accept-Encoding
< x-cache: MISS
< x-consul-default-acl-policy: deny
< x-consul-index: 486
...

{
  "SerialNumber": "59:3e:c6:7f:5c:2c:ac:d8:bb:50:d3:87:ec:3b:41:44:1d:9a:86:82",
  "CertPEM": "-----BEGIN CERTIFICATE----- ... -----END CERTIFICATE-----",
  "PrivateKeyPEM": "-----BEGIN EC PRIVATE KEY----- ... -----END EC PRIVATE KEY-----\n",
  "Service": "web",
  "ServiceURI": "spiffe://ec52b326-9aa4-5d2f-2032-c77aac1499b5.consul/ns/default/dc/dc1/svc/web",
  "ValidAfter": "2021-07-25T13:39:07Z",
  "ValidBefore": "2021-07-25T14:39:37Z",
  "CreateIndex": 486,
  "ModifyIndex": 486
}

This time you can verify that the x-cache value is MISS, meaning that the result is not coming from the cache, and you can verify from the output that the certificates returned are different from the ones returned by the cached request.

»Certificate rotation in logs

You can verify the rotation is successfully happening also from Consul logs.

»Server agents

When intermediate certificates for the service mesh get rotated, you will get that signaled on the leader's log.

[INFO]  agent.server.connect: generated new intermediate certificate for primary datacenter
[INFO]  agent.server.connect: updated root certificates from primary datacenter

[INFO]  agent.server.connect: generated new intermediate certificate for primary datacenter
[INFO]  agent.server.connect: updated root certificates from primary datacenter

»Client agents

Client agents will get log lines for the rotation of intermediate certificates and the rotation of the service leaf certificate. The log level for those events are DEBUG, so you have to set up that verbosity level to troubleshoot the certificate rotation.

Intermediate certificate rotation:

[DEBUG] agent.auto_config: handling a cache update event: correlation_id=roots
[DEBUG] agent.auto_config: roots watch fired - updating CA certificates

[DEBUG] agent.auto_config: handling a cache update event: correlation_id=roots
[DEBUG] agent.auto_config: roots watch fired - updating CA certificates

Leaf certificate rotation:

[DEBUG] agent.auto_config: handling a cache update event: correlation_id=leaf
[DEBUG] agent.auto_config: leaf certificate watch fired - updating TLS certificate

[DEBUG] agent.auto_config: handling a cache update event: correlation_id=leaf
[DEBUG] agent.auto_config: leaf certificate watch fired - updating TLS certificate

»Next Steps

In this tutorial, you configured Consul service mesh's certification Authority using the PKI secrets engine in Vault. You configured a root certificate for Consul service mesh backed by Vault, and generated leaf certificates for services deployed to Consul, signed by this root certificate.

Continue learning about the options of securing Consul by taking additional tutorials on Learn. The Secure Consul Gossip Communication with Vault tutorial teaches you to secure gossip protocol communication inside your Consul clusters. The Generate mTLS Certificates for Consul using Vault tutorial helps you implement mutual TLS for the overall Consul cluster.

Learn more about ACL token administration on Vault with the Generate Consul Tokens with HashiCorp Vault tutorial.

Infrastructure

Security

Applications

Networking

cloud