Skip to main content
June 20-22 Announcing HashiConf Europe full schedule: keynotes, sessions, labs & more Register Now
Type '/' to Search
Enterprise

Vault as Consul Connect Service Mesh Certification Authority

This tutorial also appears in: Secrets Management and HashiCorp Products.

You can use Vault's PKI Secrets Engine to generate and renew dynamic X.509 certificates. Aside from being useful to secure Consul RPC and Raft communication, the PKI secrets engine can also act as a Certificate Authority (CA) for Consul Connect service mesh.

One of the benefits of using a service mesh is to have certificates for the service instances generated and rotated automatically and used to authenticate and secure service communication. Using Vault as CA for your service mesh allows you to use unified workflow for certificate generation and have a unified control over all Consul certificates' lifecycle.

In this tutorial, you will learn how to configure your Consul datacenter to use Vault as the service mesh CA for new and existing data centers. This tutorial gives you two paths: Using self-managed Consul and Vault cluster. Or using Consul and Vault clusters deployed to HashiCorp cloud.

»Prerequisites

This tutorial is divided into sections: One section for readers using self-managed Vault and Consul clusters. The other section is for readers wanting, or already using Consul and Vault hosted on HashiCorp Cloud (HCP)

»Self Managed

»HashiCorp Cloud (HCP)

In this tutorial you will configure the Consul Connect Certificate Authority in two setups: First, you will deploy HCP Consul and configure the HCP Consul Connect CA with HCP Vault after the cluster has deployed, changing the Certificate Authority after the HCP Consul cluster is running.

Then, you will deploy a new Consul Enterprise cluster to Amazon Elastic Kubernetes Service (EKS) as a client to the HCP Consul cluster. You will configure the same HCP Vault cluster as the Consul Connect CA for this Consul Cluster.

In this configuration, HCP Consul Connect uses HCP Vault's root certificate for identity verification. The client cluster on EKS will use this root certificate to register and identify with HCP Consul acting as the server. The HCP Consul instance signs an intermediate certificate for the client cluster on EKS. This intermediate certificate signs leaf certificates for services deployed to the service mesh on the client cluster. You will then deploy the HashiCups application to the Consul Connect service mesh on EKS.

Reference the following diagram for the services you will deploy in this tutorial.

Diagram of HCP Consul and HCP Vault deployed to HCP on the left, a Consul Client cluster on the right deployed to Amazon EKS

A repository including terraform code for this tutorial is provided to complete your tasks. The terraform code deploys an HCP Consul Cluster, an HCP Vault Cluster, an Amazon EKS Cluster, and the underlying networking components for HCP and AWS to communicate with each other. These resources accrue charges in your HCP and AWS accounts. A cleanup script is provided at the end of this tutorial.

If you do not already have a HashiCorp Cloud account, sign up at the HashiCorp Cloud registration page. New accounts receive $50 in account credit, an amount sufficient for the HCP resources deployed during this tutorial.

NOTE: HCP Credits do not cover the associated cost of the AWS resources deployed in this tutorial.

To begin, clone the repository.

HTTPS
$ git clone https://github.com/hashicorp/learn-consul-kubernetes.git

$ git clone https://github.com/hashicorp/learn-consul-kubernetes.git

$ git clone git@github.com:hashicorp/learn-consul-kubernetes.git

$ git clone git@github.com:hashicorp/learn-consul-kubernetes.git

Navigate into the repository folder.

$ cd learn-consul-kubernetes

$ cd learn-consul-kubernetes

Fetch the tags from the git remote server, and checkout the tag for this tutorial.

$ git fetch --all --tags && git checkout tags/v0.0.16

$ git fetch --all --tags && git checkout tags/v0.0.16

Navigate into the project folder for this tutorial.

$ cd hcp-consul-hcp-vault-ca

$ cd hcp-consul-hcp-vault-ca

Place your Consul Enterprise license into the folder, from wherever you currently store it, and name it consul.hclic.

$ cp /path/to/your/current/consul-ent/license consul.hclic

$ cp /path/to/your/current/consul-ent/license consul.hclic

Set your HCP service principal credentials as environment variables.

$ export HCP_CLIENT_ID=YOUR_HCP_CLIENT_ID_GOES_HERE

$ export HCP_CLIENT_ID=YOUR_HCP_CLIENT_ID_GOES_HERE

$ export HCP_CLIENT_SECRET=YOUR_HCP_CLIENT_SECRET_GOES_HERE

$ export HCP_CLIENT_SECRET=YOUR_HCP_CLIENT_SECRET_GOES_HERE

The terraform project consists of two deployment steps. The first terraform init and apply deploys the HCP infrastructure and Amazon EKS. The second terraform init and apply creates a pod inside Amazon EKS from data generated from the first terraform project, for you to use during the tutorial.

Initialize the first terraform project.

$ terraform init

Terraform has been successfully initialized!

You may now begin working with Terraform. Try running "terraform plan" to see
any changes that are required for your infrastructure. All Terraform commands
should now work.

If you ever set or change modules or backend configuration for Terraform,
rerun this command to reinitialize your working directory. If you forget, other
commands will detect it and remind you to do so if necessary.

$ terraform init

Terraform has been successfully initialized!

You may now begin working with Terraform. Try running "terraform plan" to see
any changes that are required for your infrastructure. All Terraform commands
should now work.

If you ever set or change modules or backend configuration for Terraform,
rerun this command to reinitialize your working directory. If you forget, other
commands will detect it and remind you to do so if necessary.

Run terraform apply to deploy the project, which presents a confirmation screen to deploy the bulk of the resources for this tutorial. Type "yes", to deploy all the resources. Then, run terraform init and apply once again inside the inside the working-environment directory to deploy Kubernetes specific resources for you to complete the tasks in this tutorial.

$ terraform apply && cd working-environment; terraform init && terraform apply -auto-approve && cd ../

##...

Plan: 77 to add, 0 to change, 0 to destroy.

Changes to Outputs:
+ consul_cert_data = (sensitive value)
+ eks_cluster_host = (known after apply)
+ eks_data         = (sensitive value)
+ vault_data       = (sensitive value)

Do you want to perform these actions?
Terraform will perform the actions described above.
Only 'yes' will be accepted to approve.

Enter a value: yes

$ terraform apply && cd working-environment; terraform init && terraform apply -auto-approve && cd ../

##...

Plan: 77 to add, 0 to change, 0 to destroy.

Changes to Outputs:
+ consul_cert_data = (sensitive value)
+ eks_cluster_host = (known after apply)
+ eks_data         = (sensitive value)
+ vault_data       = (sensitive value)

Do you want to perform these actions?
Terraform will perform the actions described above.
Only 'yes' will be accepted to approve.

Enter a value: yes

The deployment takes up to 40 minutes to complete. During this time, check out this Community Office Hours recording which discusses setting up Vault as a certificate authority for a Consul cluster.

»Your workbench

You will access HCP Vault and HCP Consul from inside the AWS EKS Cluster deployed with terraform. A pod is provided for you with the tools to complete the tasks in this tutorial. Going forward, this pod is referred to as your "workbench." When workbench is referenced, it specifically means the environment inside your working environment pod.

The workbench on the deployed EKS cluster provides three pre-installed text editors: nano, emacs, and vim. If you are unfamiliar with these editors, or have a preference for another editor, the yum package manager is available for you to install packages on the container.

The workbench is available as soon as terraform apply finishes.

Save the identifier of the workbench as a shell variable, by searching for the workbench's app label.

$ export WORKBENCH=$(kubectl get pods -l app=tutorial  -o json | jq -r ".items[0].metadata.name")

$ export WORKBENCH=$(kubectl get pods -l app=tutorial  -o json | jq -r ".items[0].metadata.name")

Copy your Consul Enterprise license to the workbench.

$ kubectl cp ./consul.hclic default/"${WORKBENCH}":/tmp

$ kubectl cp ./consul.hclic default/"${WORKBENCH}":/tmp

Login to the workbench's bash shell.

$ kubectl exec -it "${WORKBENCH}" /bin/bash

$ kubectl exec -it "${WORKBENCH}" /bin/bash

The workbench, deployed on the EKS cluster, is where you will complete most of the tasks for this tutorial.

The workbench has automatically configured an IAM Role for the preinstalled AWS CLI. You will be able to access all AWS EKS resources needed during this tutorial.

List the pods in the EKS cluster with kubectl confirming EKS resources are available inside the workbench.

$ kubectl get pods --all-namespaces

NAMESPACE     NAME                        READY   STATUS    RESTARTS   AGE
default       tutorial-7759c4cd45-qxjwz   1/1     Running   0          44m
kube-system   aws-node-969ch              1/1     Running   0          4h31m
kube-system   aws-node-bm7rj              1/1     Running   0          4h31m
kube-system   aws-node-d7ctf              1/1     Running   0          4h31m
kube-system   coredns-66cb55d4f4-rpg9l    1/1     Running   0          4h37m
kube-system   coredns-66cb55d4f4-t4c4c    1/1     Running   0          4h37m
kube-system   kube-proxy-b2mnv            1/1     Running   0          4h31m
kube-system   kube-proxy-ggkws            1/1     Running   0          4h31m
kube-system   kube-proxy-hrmjt            1/1     Running   0          4h31m

$ kubectl get pods --all-namespaces

NAMESPACE     NAME                        READY   STATUS    RESTARTS   AGE
default       tutorial-7759c4cd45-qxjwz   1/1     Running   0          44m
kube-system   aws-node-969ch              1/1     Running   0          4h31m
kube-system   aws-node-bm7rj              1/1     Running   0          4h31m
kube-system   aws-node-d7ctf              1/1     Running   0          4h31m
kube-system   coredns-66cb55d4f4-rpg9l    1/1     Running   0          4h37m
kube-system   coredns-66cb55d4f4-t4c4c    1/1     Running   0          4h37m
kube-system   kube-proxy-b2mnv            1/1     Running   0          4h31m
kube-system   kube-proxy-ggkws            1/1     Running   0          4h31m
kube-system   kube-proxy-hrmjt            1/1     Running   0          4h31m

The workbench automatically pre-loads the HashiCorp Cloud tokens necessary to communicate with the HashiCorp Cloud (HCP) resources. Verify HCP Vault is available by using Vault's status command.

$ vault status

Key                      Value
---                      -----
Recovery Seal Type       shamir
Initialized              true
Sealed                   false
Total Recovery Shares    1
Threshold                1
Version                  1.9.2+ent
Storage Type             raft
Cluster Name             vault-cluster-6c58fbd8
Cluster ID               91fd15b1-f17f-36b1-abd3-75c3a438ed2b
HA Enabled               true
HA Cluster               https://10.100.7.212:8201
HA Mode                  active
Active Since             2022-02-15T21:45:48.63677326Z
Raft Committed Index     4832
Raft Applied Index       4832
Last WAL                 710

$ vault status

Key                      Value
---                      -----
Recovery Seal Type       shamir
Initialized              true
Sealed                   false
Total Recovery Shares    1
Threshold                1
Version                  1.9.2+ent
Storage Type             raft
Cluster Name             vault-cluster-6c58fbd8
Cluster ID               91fd15b1-f17f-36b1-abd3-75c3a438ed2b
HA Enabled               true
HA Cluster               https://10.100.7.212:8201
HA Mode                  active
Active Since             2022-02-15T21:45:48.63677326Z
Raft Committed Index     4832
Raft Applied Index       4832
Last WAL                 710

Verify HCP Consul is available by using Consul's members command.

$ consul members

  Node             Address            Status  Type    Build       Protocol  DC   Partition  Segment
  ip-10-100-20-55  10.100.20.55:8301  alive   server  1.11.3+ent  2         dc1  default    all

$ consul members

  Node             Address            Status  Type    Build       Protocol  DC   Partition  Segment
  ip-10-100-20-55  10.100.20.55:8301  alive   server  1.11.3+ent  2         dc1  default    all

»HCP Consul Connect Certificate Authority

When Consul Connect is initially enabled on HCP Consul, it self-signs its own TLS certificates. In this section, you are configuring HCP Consul to use HCP Vault as the Certificate Authority.

While an HCP Consul Cluster is active, configuration changes to the HCP Consul service mesh Certificate Authority are actively ignored. This includes when an HCP Consul instance is reloaded. Or if the underlying HCP hosts are restarted. Instead, Consul service mesh provides a specific API endpoint and Consul Connect CLI command to update the certificate authority while the Consul cluster is active.

To observe HCP Consul service mesh's current self-signing status, use the CLI or Consul REST API to return the CA configuration.

Consul CLI
$ consul connect ca get-config

    {
      "Provider": "consul",
      "Config": {
      "IntermediateCertTTL": "8760h",
      "LeafCertTTL": "72h",
      "RootCertTTL": "87600h"
    },
      "State": null,
      "ForceWithoutCrossSigning": false,
      "CreateIndex": 9,
      "ModifyIndex": 9
    }


$ consul connect ca get-config

    {
      "Provider": "consul",
      "Config": {
      "IntermediateCertTTL": "8760h",
      "LeafCertTTL": "72h",
      "RootCertTTL": "87600h"
    },
      "State": null,
      "ForceWithoutCrossSigning": false,
      "CreateIndex": 9,
      "ModifyIndex": 9
    }


$ curl --header "X-Consul-Token: $CONSUL_HTTP_TOKEN" $CONSUL_HTTP_ADDR/v1/connect/ca/configuration | jq

  {
    "Provider": "consul",
    "Config": {
    "IntermediateCertTTL": "8760h",
    "LeafCertTTL": "72h",
    "RootCertTTL": "87600h"
     },
    "State": null,
    "ForceWithoutCrossSigning": false,
    "CreateIndex": 9,
    "ModifyIndex": 9
  }

$ curl --header "X-Consul-Token: $CONSUL_HTTP_TOKEN" $CONSUL_HTTP_ADDR/v1/connect/ca/configuration | jq

  {
    "Provider": "consul",
    "Config": {
    "IntermediateCertTTL": "8760h",
    "LeafCertTTL": "72h",
    "RootCertTTL": "87600h"
     },
    "State": null,
    "ForceWithoutCrossSigning": false,
    "CreateIndex": 9,
    "ModifyIndex": 9
  }

»Manage HCP Vault pki secrets engine paths

In this section you will begin to configure HCP Vault as the HCP Consul Connect certificate authority. You will create an HCP Vault secrets engine for the root certificate in the /connect_root pki path, and a leaf secrets engine, in the /connect_inter pki path, both in HCP Vault.

The root certificates is hosted in HCP Vault, and managed by HCP Consul. The root certificate is used to verify the identity of the clusters, and to sign intermediate certificates, also known as leaf certificates. Leaf certificates are used for services inside the Consul Connect service mesh.

In the following example, the Vault paths represent the Consul service mesh root authority as connect_root. The intermediate authority path is represented as connect_inter. These values are customizable if you have alternative requirements in non-tutorial environments.

Create the HCP Consul Connect policy file.

$ touch vault-policy-connect-ca.hcl

$ touch vault-policy-connect-ca.hcl

Add the following configuration to the policy file.

    # Consul Managed PKI Mounts
    path "/sys/mounts" {
    capabilities = [ "read" ]
  }

    path "/sys/mounts/connect_root" {
    capabilities = [ "create", "read", "update", "delete", "list" ]
  }

    path "/sys/mounts/connect_inter" {
    capabilities = [ "create", "read", "update", "delete", "list" ]
  }

    path "connect_root/*" {
    capabilities = [ "create", "read", "update", "delete", "list" ]
  }

    path "connect_inter/*" {
    capabilities = [ "create", "read", "update", "delete", "list" ]
  }
vault-policy-connect-ca.hcl
    # Consul Managed PKI Mounts
    path "/sys/mounts" {
    capabilities = [ "read" ]
  }

    path "/sys/mounts/connect_root" {
    capabilities = [ "create", "read", "update", "delete", "list" ]
  }

    path "/sys/mounts/connect_inter" {
    capabilities = [ "create", "read", "update", "delete", "list" ]
  }

    path "connect_root/*" {
    capabilities = [ "create", "read", "update", "delete", "list" ]
  }

    path "connect_inter/*" {
    capabilities = [ "create", "read", "update", "delete", "list" ]
  }

Implement the policy definition using the HCP Vault CLI.

$ vault policy write connect-ca vault-policy-connect-ca.hcl

Success! Uploaded policy: connect-ca

$ vault policy write connect-ca vault-policy-connect-ca.hcl

Success! Uploaded policy: connect-ca

»Create HCP Vault ACL token

After the policy is created, generate the HCP Vault policy token HCP Consul uses to access and manage the PKI secrets in HCP Vault. When using HCP Vault, you are required to specify the Vault namespace where this token is scoped. This tutorial uses the primary HCP Vault namespace admin.

Create the Consul ACL token with the Vault CLI, using the connect-ca policy you created in the previous section, and saving it as an environment variable.

$ RESPONSE=$(vault token create -period=168h -policy=connect-ca -namespace=admin -format=json); echo "$RESPONSE" && \
  export VAULT_CONSUL_CONNECT_CA_TOKEN=$(jq -r '.auth.client_token' <<< $RESPONSE)

$ RESPONSE=$(vault token create -period=168h -policy=connect-ca -namespace=admin -format=json); echo "$RESPONSE" && \
  export VAULT_CONSUL_CONNECT_CA_TOKEN=$(jq -r '.auth.client_token' <<< $RESPONSE)

{
  "request_id": "8b89bdca-2df8-659c-2f1f-a9b0bc49875f",
  "lease_id": "",
  "lease_duration": 0,
  "renewable": false,
  "data": null,
  "warnings": null,
  "auth": {
  "client_token": "s.jGZhtW28R1hrcP7FWXA7d3Tp.zFhPO",
  "accessor": "BYGlDSOCLLwtIzCrnjvwRb3z.zFhPO",
  "policies": [
  "connect-ca",
  "default"
  ],
  "token_policies": [
  "connect-ca",
  "default"
  ],
  "identity_policies": null,
  "metadata": null,
  "orphan": false,
  "entity_id": "42fa2184-fd7b-b3d1-3e2e-66d7343bf65f",
  "lease_duration": 604800,
  "renewable": true
  }
}

{
  "request_id": "8b89bdca-2df8-659c-2f1f-a9b0bc49875f",
  "lease_id": "",
  "lease_duration": 0,
  "renewable": false,
  "data": null,
  "warnings": null,
  "auth": {
  "client_token": "s.jGZhtW28R1hrcP7FWXA7d3Tp.zFhPO",
  "accessor": "BYGlDSOCLLwtIzCrnjvwRb3z.zFhPO",
  "policies": [
  "connect-ca",
  "default"
  ],
  "token_policies": [
  "connect-ca",
  "default"
  ],
  "identity_policies": null,
  "metadata": null,
  "orphan": false,
  "entity_id": "42fa2184-fd7b-b3d1-3e2e-66d7343bf65f",
  "lease_duration": 604800,
  "renewable": true
  }
}

Create the Consul Connect CA configuration file.

$ touch config-connect-ca-provider-vault.json

$ touch config-connect-ca-provider-vault.json

Output the environment variables to the terminal, validating they are not empty.

$ echo "Vault connect token: $VAULT_CONSUL_CONNECT_CA_TOKEN" && \
  echo "Vault Address: $VAULT_ADDR" && \
  echo "Vault Namespace: $VAULT_NAMESPACE"

$ echo "Vault connect token: $VAULT_CONSUL_CONNECT_CA_TOKEN" && \
  echo "Vault Address: $VAULT_ADDR" && \
  echo "Vault Namespace: $VAULT_NAMESPACE"

Vault Connect CA ACL token: s.B1E36IpNp8kKL7msz8pPmZMi.zFhPO
Vault Address: https://vault-cluster.private.vault.223350f5-f0b9-4d89-959d-aa8eab3184e5.aws.hashicorp.cloud:8200
Vault Namespace: admin

Vault Connect CA ACL token: s.B1E36IpNp8kKL7msz8pPmZMi.zFhPO
Vault Address: https://vault-cluster.private.vault.223350f5-f0b9-4d89-959d-aa8eab3184e5.aws.hashicorp.cloud:8200
Vault Namespace: admin

The root and Intermediate PKI path values need to match the paths set for the Vault ACL token policy you created previously in this tutorial. Copy the configuration below into the configuration file. Replace the environment variable placeholders with the corresponding shell output, as displayed in the echo statement above. For example, $VAULT_NAMESPACE is replaced with admin.

{
  "Provider": "vault",
  "Config": {
  "Address": "$VAULT_ADDR",
  "Token": "$VAULT_CONSUL_CONNECT_CA_TOKEN",
  "RootPKIPath": "connect_root",
  "IntermediatePKIPath": "connect_inter",
  "LeafCertTTL": "72h",
  "RotationPeriod": "2160h",
  "IntermediateCertTTL": "8760h",
  "PrivateKeyType": "rsa",
  "PrivateKeyBits": 2048,
  "Namespace": "$VAULT_NAMESPACE"
  },
  "ForceWithoutCrossSigning": false
}
config-connect-ca-provider-vault.json
1 2 3 4 5 6 7 8 9 10111213141516{
  "Provider": "vault",
  "Config": {
  "Address": "$VAULT_ADDR",
  "Token": "$VAULT_CONSUL_CONNECT_CA_TOKEN",
  "RootPKIPath": "connect_root",
  "IntermediatePKIPath": "connect_inter",
  "LeafCertTTL": "72h",
  "RotationPeriod": "2160h",
  "IntermediateCertTTL": "8760h",
  "PrivateKeyType": "rsa",
  "PrivateKeyBits": 2048,
  "Namespace": "$VAULT_NAMESPACE"
  },
  "ForceWithoutCrossSigning": false
}

Implement this HCP Consul configuration with the Consul Connect CLI.

$ consul connect ca set-config -config-file=./config-connect-ca-provider-vault.json


Configuration updated!

$ consul connect ca set-config -config-file=./config-connect-ca-provider-vault.json


Configuration updated!

»Verify the HCP Consul Connect Certificate Authority has changed

After implementing the configuration, the certificate authority immediately updates. To confirm and observe, use the Consul Connect CLI, or REST API to query the configuration for Consul Connect CA.

Consul CLI
$ consul connect ca get-config

{
  "Provider": "vault",
  "Config": {
  "Address": "https://vault-cluster.private.vault.223350f5-f0b9-4d89-959d-aa8eab3184e5.aws.hashicorp.cloud:8200",
  "IntermediateCertTTL": "8760h",
  "IntermediatePKIPath": "connect_inter",
  "LeafCertTTL": "72h",
  "Namespace": "admin",
  "PrivateKeyBits": 2048,
  "PrivateKeyType": "rsa",
  "RootPKIPath": "connect_root",
  "RotationPeriod": "2160h",
  "Token": "s.P7XuID9v2he5wksP62GHKprX.8NzGx"
},
  "State": null,
  "ForceWithoutCrossSigning": false,
  "CreateIndex": 9,
  "ModifyIndex": 1208
}

$ consul connect ca get-config

{
  "Provider": "vault",
  "Config": {
  "Address": "https://vault-cluster.private.vault.223350f5-f0b9-4d89-959d-aa8eab3184e5.aws.hashicorp.cloud:8200",
  "IntermediateCertTTL": "8760h",
  "IntermediatePKIPath": "connect_inter",
  "LeafCertTTL": "72h",
  "Namespace": "admin",
  "PrivateKeyBits": 2048,
  "PrivateKeyType": "rsa",
  "RootPKIPath": "connect_root",
  "RotationPeriod": "2160h",
  "Token": "s.P7XuID9v2he5wksP62GHKprX.8NzGx"
},
  "State": null,
  "ForceWithoutCrossSigning": false,
  "CreateIndex": 9,
  "ModifyIndex": 1208
}

$ curl --verbose --header "X-Consul-Token: $CONSUL_HTTP_TOKEN" ${CONSUL_HTTP_ADDR}/v1/agent/connect/ca/roots | jq


* Connected to dc1.private.consul.223350f5-f0b9-4d89-959d-aa8eab3184e5.aws.hashicorp.cloud (10.100.0.58) port 443 (#0)
* SSL connection using TLSv1.2 / ECDHE-RSA-AES128-GCM-SHA256
* ALPN, server accepted to use h2
* Server certificate:
*  subject: CN=df4455cc-5adc-43e9-b945-e15fbdc09789.aws.hashicorp.cloud
*  start date: Feb 24 14:20:51 2022 GMT
*  expire date: May 25 14:20:50 2022 GMT
*  subjectAltName: host "dc1.private.consul.223350f5-f0b9-4d89-959d-aa8eab3184e5.aws.hashicorp.cloud" matched cert's "dc1.private.consul.223350f5-f0b9-4d89-959d-aa8eab3184e5.aws.hashicorp.cloud"
*  issuer: C=US; O=Let's Encrypt; CN=R3
*  SSL certificate verify ok.


> GET /v1/agent/connect/ca/roots HTTP/2
> Host: dc1.private.consul.223350f5-f0b9-4d89-959d-aa8eab3184e5.aws.hashicorp.cloud
> user-agent: curl/7.79.1
> accept: */*
> x-consul-token: 706e21ce-cf5b-54ae-beb1-0f2ca536bc36
< HTTP/2 200
< age: 0
< content-type: application/json
< date: Thu, 24 Feb 2022 20:45:57 GMT
< strict-transport-security: max-age=31536000
< vary: Accept-Encoding
< x-cache: HIT
< x-consul-default-acl-policy: deny
< x-consul-index: 1208
< x-consul-knownleader: true
< x-consul-lastcontact: 0
< x-consul-query-backend: blocking-query
<

* Connection #0 to host dc1.private.consul.223350f5-f0b9-4d89-959d-aa8eab3184e5.aws.hashicorp.cloud left intact
{
  "ActiveRootID": "c1:ee:2f:93:0f:4a:05:23:fb:c3:7c:ca:6a:d7:90:31:64:de:a6:b0",
  "TrustDomain": "89b781a1-5172-e0c0-3241-e62cbf768e6c.consul",
  "Roots": [
{
  "ID": "c1:ee:2f:93:0f:4a:05:23:fb:c3:7c:ca:6a:d7:90:31:64:de:a6:b0",
  "Name": "Vault CA Root Cert",
  "SerialNumber": 9312715522156884000,
  "SigningKeyID": "47:d2:6c:c9:cc:0e:13:b4:e5:de:bd:bc:98:e3:88:4c:8a:fa:3f:2f",
  "ExternalTrustDomain": "89b781a1-5172-e0c0-3241-e62cbf768e6c",
  "NotBefore": "2022-02-24T20:35:19Z",
  "NotAfter": "2032-02-22T20:35:48Z",
  "RootCert": "-----BEGIN CERTIFICATE----- Edited for Brevity-----END CERTIFICATE-----\n",
  "IntermediateCerts": [
  "-----BEGIN CERTIFICATE----- Edited for Brevity ----END CERTIFICATE-----\n",
  "-----BEGIN CERTIFICATE----- Edited for Brevity -----END CERTIFICATE-----\n"
  ],
  "Active": true,
  "PrivateKeyType": "rsa",
  "PrivateKeyBits": 2048,
  "CreateIndex": 1208,
  "ModifyIndex": 1208
},
{
  "ID": "eb:d3:94:cc:56:5c:f2:be:15:ec:2d:79:69:f0:6c:6e:3c:46:16:1f",
  "Name": "Consul CA Root Cert",
  "SerialNumber": 11,
  "SigningKeyID": "aa:41:a5:37:11:14:01:97:fa:e7:ed:44:28:06:23:78:f5:34:59:dc:3a:64:da:9c:9e:aa:41:14:f0:6b:c5:91",
  "ExternalTrustDomain": "89b781a1-5172-e0c0-3241-e62cbf768e6c",
  "NotBefore": "2022-02-24T15:23:40Z",
  "NotAfter": "2032-02-22T15:23:40Z",
  "RootCert": "-----BEGIN CERTIFICATE----- Edited for Brevity ----END CERTIFICATE-----\n",
  "IntermediateCerts": null,
  "Active": false,
  "PrivateKeyType": "ec",
  "PrivateKeyBits": 256,
  "CreateIndex": 13,
  "ModifyIndex": 1208
}
  ]
}


$ curl --verbose --header "X-Consul-Token: $CONSUL_HTTP_TOKEN" ${CONSUL_HTTP_ADDR}/v1/agent/connect/ca/roots | jq


* Connected to dc1.private.consul.223350f5-f0b9-4d89-959d-aa8eab3184e5.aws.hashicorp.cloud (10.100.0.58) port 443 (#0)
* SSL connection using TLSv1.2 / ECDHE-RSA-AES128-GCM-SHA256
* ALPN, server accepted to use h2
* Server certificate:
*  subject: CN=df4455cc-5adc-43e9-b945-e15fbdc09789.aws.hashicorp.cloud
*  start date: Feb 24 14:20:51 2022 GMT
*  expire date: May 25 14:20:50 2022 GMT
*  subjectAltName: host "dc1.private.consul.223350f5-f0b9-4d89-959d-aa8eab3184e5.aws.hashicorp.cloud" matched cert's "dc1.private.consul.223350f5-f0b9-4d89-959d-aa8eab3184e5.aws.hashicorp.cloud"
*  issuer: C=US; O=Let's Encrypt; CN=R3
*  SSL certificate verify ok.


> GET /v1/agent/connect/ca/roots HTTP/2
> Host: dc1.private.consul.223350f5-f0b9-4d89-959d-aa8eab3184e5.aws.hashicorp.cloud
> user-agent: curl/7.79.1
> accept: */*
> x-consul-token: 706e21ce-cf5b-54ae-beb1-0f2ca536bc36
< HTTP/2 200
< age: 0
< content-type: application/json
< date: Thu, 24 Feb 2022 20:45:57 GMT
< strict-transport-security: max-age=31536000
< vary: Accept-Encoding
< x-cache: HIT
< x-consul-default-acl-policy: deny
< x-consul-index: 1208
< x-consul-knownleader: true
< x-consul-lastcontact: 0
< x-consul-query-backend: blocking-query
<

* Connection #0 to host dc1.private.consul.223350f5-f0b9-4d89-959d-aa8eab3184e5.aws.hashicorp.cloud left intact
{
  "ActiveRootID": "c1:ee:2f:93:0f:4a:05:23:fb:c3:7c:ca:6a:d7:90:31:64:de:a6:b0",
  "TrustDomain": "89b781a1-5172-e0c0-3241-e62cbf768e6c.consul",
  "Roots": [
{
  "ID": "c1:ee:2f:93:0f:4a:05:23:fb:c3:7c:ca:6a:d7:90:31:64:de:a6:b0",
  "Name": "Vault CA Root Cert",
  "SerialNumber": 9312715522156884000,
  "SigningKeyID": "47:d2:6c:c9:cc:0e:13:b4:e5:de:bd:bc:98:e3:88:4c:8a:fa:3f:2f",
  "ExternalTrustDomain": "89b781a1-5172-e0c0-3241-e62cbf768e6c",
  "NotBefore": "2022-02-24T20:35:19Z",
  "NotAfter": "2032-02-22T20:35:48Z",
  "RootCert": "-----BEGIN CERTIFICATE----- Edited for Brevity-----END CERTIFICATE-----\n",
  "IntermediateCerts": [
  "-----BEGIN CERTIFICATE----- Edited for Brevity ----END CERTIFICATE-----\n",
  "-----BEGIN CERTIFICATE----- Edited for Brevity -----END CERTIFICATE-----\n"
  ],
  "Active": true,
  "PrivateKeyType": "rsa",
  "PrivateKeyBits": 2048,
  "CreateIndex": 1208,
  "ModifyIndex": 1208
},
{
  "ID": "eb:d3:94:cc:56:5c:f2:be:15:ec:2d:79:69:f0:6c:6e:3c:46:16:1f",
  "Name": "Consul CA Root Cert",
  "SerialNumber": 11,
  "SigningKeyID": "aa:41:a5:37:11:14:01:97:fa:e7:ed:44:28:06:23:78:f5:34:59:dc:3a:64:da:9c:9e:aa:41:14:f0:6b:c5:91",
  "ExternalTrustDomain": "89b781a1-5172-e0c0-3241-e62cbf768e6c",
  "NotBefore": "2022-02-24T15:23:40Z",
  "NotAfter": "2032-02-22T15:23:40Z",
  "RootCert": "-----BEGIN CERTIFICATE----- Edited for Brevity ----END CERTIFICATE-----\n",
  "IntermediateCerts": null,
  "Active": false,
  "PrivateKeyType": "ec",
  "PrivateKeyBits": 256,
  "CreateIndex": 13,
  "ModifyIndex": 1208
}
  ]
}

»Configure HCP Vault to HCP Consul Connection

To connect to HCP Vault in Kubernetes, the Kubernetes cluster needs a Vault agent installed on the cluster to pass the stored certificates to the Consul client cluster.

»Install Vault agent on EKS cluster

Using the vault-k8s helm chart, deploy a Vault agent to the cluster.

Before deploying Consul onto Kubernetes and connecting to HCP Consul, a Vault agent is installed on your EKS cluster, serving as the link between HCP Vault and the Consul client cluster. The agent retrieves certificates for the Consul client cluster when requested.

Create a values file for the Vault agent, and output your HCP Vault's URL.

$ touch vault-agent-values.yaml && echo $VAULT_ADDR

https://vault-cluster.private.vault.223350f5-f0b9-4d89-959d-aa8eab3184e5.aws.hashicorp.cloud:8200

$ touch vault-agent-values.yaml && echo $VAULT_ADDR

https://vault-cluster.private.vault.223350f5-f0b9-4d89-959d-aa8eab3184e5.aws.hashicorp.cloud:8200

Replace $VAULT_ADDR with the contents of the $VAULT_ADDR environment variable in your workbench. In this tutorial, it is the private URL for the HCP Vault cluster. Copy the following configuration into the values file.

  global:
    enabled: true
  injector:
    enabled: true
    externalVaultAddr: $VAULT_ADDR
  server:
    enabled: false
vault-agent-values.yaml
12345678  global:
    enabled: true
  injector:
    enabled: true
    externalVaultAddr: $VAULT_ADDR
  server:
    enabled: false

Add the Hashicorp repository to helm, to install the Vault agent on EKS.

$ helm repo add hashicorp https://helm.releases.hashicorp.com && helm repo update

  "hashicorp" has been added to your repositories
  Hang tight while we grab the latest from your chart repositories...
  ...Successfully got an update from the "hashicorp" chart repository
  Update Complete. ⎈Happy Helming!


$ helm repo add hashicorp https://helm.releases.hashicorp.com && helm repo update

  "hashicorp" has been added to your repositories
  Hang tight while we grab the latest from your chart repositories...
  ...Successfully got an update from the "hashicorp" chart repository
  Update Complete. ⎈Happy Helming!

Deploy the Vault agent to Kubernetes.

$ helm install vault --values vault-agent-values.yaml hashicorp/vault

  NAME: vault
  LAST DEPLOYED: Fri Feb 25 16:55:32 2022
  NAMESPACE: default
  STATUS: deployed
  REVISION: 1
  NOTES:
  Thank you for installing HashiCorp Vault!

  Now that you have deployed Vault, you should look over the docs on using
  Vault with Kubernetes available here:

  https://www.vaultproject.io/docs/


  Your release is named vault. To learn more about the release, try:

  $ helm status vault
  $ helm get manifest vault

$ helm install vault --values vault-agent-values.yaml hashicorp/vault

  NAME: vault
  LAST DEPLOYED: Fri Feb 25 16:55:32 2022
  NAMESPACE: default
  STATUS: deployed
  REVISION: 1
  NOTES:
  Thank you for installing HashiCorp Vault!

  Now that you have deployed Vault, you should look over the docs on using
  Vault with Kubernetes available here:

  https://www.vaultproject.io/docs/


  Your release is named vault. To learn more about the release, try:

  $ helm status vault
  $ helm get manifest vault

After deployment, the Vault agent searches for the $VAULT_TOKEN environment variable that is set in your environment, to create the initial connection to HCP Vault. This is an administrator token. In production environments, consider using an alternative token with limited permissions.

NOTE: The $VAULT_TOKEN has a default ttl policy of six hours. If you do not complete this tutorial within six hours, the HCP Vault agents will be unable to connect to HCP Vault, and you will need to generate a new token.

Validate the vault-agent-injector pod is in a working state with kubectl.

$ kubectl get pods

  NAME                                    READY   STATUS    RESTARTS   AGE
  tutorial-795bbc7684-8wq52               1/1     Running   0          84m
  vault-agent-injector-55555d9697-b5csw   1/1     Running   0          7m47s

$ kubectl get pods

  NAME                                    READY   STATUS    RESTARTS   AGE
  tutorial-795bbc7684-8wq52               1/1     Running   0          84m
  vault-agent-injector-55555d9697-b5csw   1/1     Running   0          7m47s

After the Vault agent is installed, a service account is created in the EKS cluster. You will use information from this service account in the next section. Validate the service account was created by using the following command.

$ kubectl get serviceaccount vault

  NAME    SECRETS   AGE
  vault   1         14m

$ kubectl get serviceaccount vault

  NAME    SECRETS   AGE
  vault   1         14m

»Configure Kubernetes auth engine for HCP Vault

You will now configure the Kubernetes auth engine on HCP Vault with the roles and policies required for the Vault agent to connect to HCP Vault.

For the EKS Client Cluster to retrieve secrets and TLS certificates from HCP Vault, enable the Kubernetes auth engine, and the kv secrets engine using the Vault CLI.

$ vault auth enable kubernetes && vault secrets enable -path=kv kv-v2

Success! Enabled kubernetes auth method at: kubernetes/

Success! Enabled the kv-v2 secrets engine at: kv

$ vault auth enable kubernetes && vault secrets enable -path=kv kv-v2

Success! Enabled kubernetes auth method at: kubernetes/

Success! Enabled the kv-v2 secrets engine at: kv

»Create HCP Vault roles and policies

Next, create a role and policy in HCP Vault that maps to the Consul service account in Kubernetes. The Consul service account does not exist yet, so you calculate the name of the service account. Next, you will save several environment variables that are needed to configure the role and policy.

Retrieve data stored in the Vault service account on Kubernetes to update the Kubernetes auth engine in HCP Vault. This adds the service account's certificate and token to the auth engine in HCP Vault, and configures the auth engine with the Kubernetes control plane URL.

Set the following environment variables in your shell environment. Update the Kube auth engine in HCP Vault using these values.

The JSON Web Token is used to authenticate the service account to HCP Vault.

$ export JWT_TOKEN_REVIEW_SECRET_NAME=$(kubectl get serviceaccount vault -o jsonpath='{.secrets[0].name}') \
  && export JWT_REVIEW_TOKEN_SECRET_CONTENTS_DECODED=$(kubectl get secret "${JWT_TOKEN_REVIEW_SECRET_NAME}" -o jsonpath='{.data.token}' | base64 --decode)

$ export JWT_TOKEN_REVIEW_SECRET_NAME=$(kubectl get serviceaccount vault -o jsonpath='{.secrets[0].name}') \
  && export JWT_REVIEW_TOKEN_SECRET_CONTENTS_DECODED=$(kubectl get secret "${JWT_TOKEN_REVIEW_SECRET_NAME}" -o jsonpath='{.data.token}' | base64 --decode)

The JWT's CA Certificate handles identity verification between HCP Consul and HCP Vault, for the Vault agent.

$ export JWT_REVIEW_TOKEN_CA_CERTIFICATE=$(kubectl get secret "${JWT_TOKEN_REVIEW_SECRET_NAME}" -o jsonpath='{.data.ca\.crt}' | base64 --decode)

$ export JWT_REVIEW_TOKEN_CA_CERTIFICATE=$(kubectl get secret "${JWT_TOKEN_REVIEW_SECRET_NAME}" -o jsonpath='{.data.ca\.crt}' | base64 --decode)

The control plane URL is used by the Vault agent to notify HCP Vault the location of the Kubernetes cluster.

$ export KUBE_CONTROL_PLANE=$(kubectl config view --raw --minify --flatten -o jsonpath='{.clusters[].cluster.server}':443)

$ export KUBE_CONTROL_PLANE=$(kubectl config view --raw --minify --flatten -o jsonpath='{.clusters[].cluster.server}':443)

Update the Kubernetes Auth Engine in HCP Vault with these environment variables.

$ vault write -namespace=admin auth/kubernetes/config \
    token_reviewer_jwt="${JWT_REVIEW_TOKEN_SECRET_CONTENTS_DECODED}" \
    kubernetes_host="${KUBE_CONTROL_PLANE}" \
    kubernetes_ca_cert="${JWT_REVIEW_TOKEN_CA_CERTIFICATE}"

$ vault write -namespace=admin auth/kubernetes/config \
    token_reviewer_jwt="${JWT_REVIEW_TOKEN_SECRET_CONTENTS_DECODED}" \
    kubernetes_host="${KUBE_CONTROL_PLANE}" \
    kubernetes_ca_cert="${JWT_REVIEW_TOKEN_CA_CERTIFICATE}"

»Configure HCP Vault role

Begin configuring the HCP Vault role.

You will pre-render a helm template for Consul on Kubernetes to calculate the Consul service account name, using this name to create an HCP Vault role and policy.

Create the name of the Consul release and the name of the HCP Vault role. Set this name as an environment variable.

Pass the release name variable to the helm template command, specifying the client-daemonset template, which creates the Consul service account for Kubernetes.

$ export CONSUL_RELEASE=consul && export CONSUL_SERVICE_ACCOUNT=$(helm template -s templates/client-daemonset.yaml $CONSUL_RELEASE hashicorp/consul | yq e '.metadata.name') && echo $CONSUL_SERVICE_ACCOUNT

consul-consul-client

$ export CONSUL_RELEASE=consul && export CONSUL_SERVICE_ACCOUNT=$(helm template -s templates/client-daemonset.yaml $CONSUL_RELEASE hashicorp/consul | yq e '.metadata.name') && echo $CONSUL_SERVICE_ACCOUNT

consul-consul-client

This command doesn't create the service account, the service account is created when you install Consul on Kubernetes later in this tutorial.

Store the JWT's CA Certificate in Vault KV.

$ vault kv put -namespace=admin kv/caCertificate key="${JWT_REVIEW_TOKEN_CA_CERTIFICATE}"

$ vault kv put -namespace=admin kv/caCertificate key="${JWT_REVIEW_TOKEN_CA_CERTIFICATE}"

Validate the existence of the certificate.

$ vault kv get -namespace=admin kv/caCertificate

======= Metadata =======
Key                Value
---                -----
created_time       2022-02-25T18:51:28.323739086Z
custom_metadata    <nil>
deletion_time      n/a
destroyed          false
version            1

=== Data ===
Key    Value
---    -----
key    -----BEGIN CERTIFICATE-----
MIIC5zCCAc+gAwIBAgIBADANBgkqhkiG9w0BAQsFADAVMRMwEQYDVQQDEwprdWJl
cm5ldGVzMB4XDTIyMDMwMzE1MzM1M1oXDTMyMDIyOTE1MzM1M1owFTETMBEGA1UE
AxMKa3ViZXJuZXRlczCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBAMSg
PIx2zHouMkL4Vd+y8LjNncJ8ivrZFt1w7Bw+02AE8EVWb8brbK08iv2UW5j8O3xI
F0xyPVyTN1EdEBwEEMsz2wFUqgv0+Q6K48fMVlwGG4D3dUxDNxXsFA7EaxEYtLlb
ikx/nU2VK4hVBkpztTyFZwupYiHhkvK6SrnN+C4xDOpBPjK627LMNIhRj/m7plwp
oH9KjJ9XYv+iaOptitwp9xLtJcw4QA56MH6CYgLXGRnbbl1Lk8e+2lBJhe8PKtac
55wY9thiyy6H3IwWiVinWABYQzscU4Xslyg65+wbXxKe87ttWpCHetwa+rlCtUHn
SvZGdaUKLe8H4sUNS0UCAwEAAaNCMEAwDgYDVR0PAQH/BAQDAgKkMA8GA1UdEwEB
/wQFMAMBAf8wHQYDVR0OBBYEFGkyGCDvFZyFHxF6WvJz1OH5qcnFMA0GCSqGSIb3
DQEBCwUAA4IBAQC8181V2V6kg+91bMhtt6f4cA3WRjRqgNvLHm7nPspFhOdN2Elh
d6U1Krsgi1KyQ/XElAfXkGbXiDf8bpCwD3oYliVwyPN7yHfEh5CuqO1uSDP6Lkml
05RD5bMY6fqBeTjDXiGSG5JdoIEeVT6Wf+5mp+fkljjpb08TSpahuvUJfSW7b5xO
4e2PuBkKwHrZ5GEWIo3onne0RK9qdc6CdYizsz1T7SbqY/2hLSRC2d7BVKIOnNBs
gcqS+mJQ3+k2UFfaQ8vZQFZV9AD84YNkHoyQxD6eLoB50K1znvC5NrZ875ii+PDI
ezvOxkpohnFwcgUHe5wtEVzy/4HvqkNlWdfw
-----END CERTIFICATE-----


$ vault kv get -namespace=admin kv/caCertificate

======= Metadata =======
Key                Value
---                -----
created_time       2022-02-25T18:51:28.323739086Z
custom_metadata    <nil>
deletion_time      n/a
destroyed          false
version            1

=== Data ===
Key    Value
---    -----
key    -----BEGIN CERTIFICATE-----
MIIC5zCCAc+gAwIBAgIBADANBgkqhkiG9w0BAQsFADAVMRMwEQYDVQQDEwprdWJl
cm5ldGVzMB4XDTIyMDMwMzE1MzM1M1oXDTMyMDIyOTE1MzM1M1owFTETMBEGA1UE
AxMKa3ViZXJuZXRlczCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBAMSg
PIx2zHouMkL4Vd+y8LjNncJ8ivrZFt1w7Bw+02AE8EVWb8brbK08iv2UW5j8O3xI
F0xyPVyTN1EdEBwEEMsz2wFUqgv0+Q6K48fMVlwGG4D3dUxDNxXsFA7EaxEYtLlb
ikx/nU2VK4hVBkpztTyFZwupYiHhkvK6SrnN+C4xDOpBPjK627LMNIhRj/m7plwp
oH9KjJ9XYv+iaOptitwp9xLtJcw4QA56MH6CYgLXGRnbbl1Lk8e+2lBJhe8PKtac
55wY9thiyy6H3IwWiVinWABYQzscU4Xslyg65+wbXxKe87ttWpCHetwa+rlCtUHn
SvZGdaUKLe8H4sUNS0UCAwEAAaNCMEAwDgYDVR0PAQH/BAQDAgKkMA8GA1UdEwEB
/wQFMAMBAf8wHQYDVR0OBBYEFGkyGCDvFZyFHxF6WvJz1OH5qcnFMA0GCSqGSIb3
DQEBCwUAA4IBAQC8181V2V6kg+91bMhtt6f4cA3WRjRqgNvLHm7nPspFhOdN2Elh
d6U1Krsgi1KyQ/XElAfXkGbXiDf8bpCwD3oYliVwyPN7yHfEh5CuqO1uSDP6Lkml
05RD5bMY6fqBeTjDXiGSG5JdoIEeVT6Wf+5mp+fkljjpb08TSpahuvUJfSW7b5xO
4e2PuBkKwHrZ5GEWIo3onne0RK9qdc6CdYizsz1T7SbqY/2hLSRC2d7BVKIOnNBs
gcqS+mJQ3+k2UFfaQ8vZQFZV9AD84YNkHoyQxD6eLoB50K1znvC5NrZ875ii+PDI
ezvOxkpohnFwcgUHe5wtEVzy/4HvqkNlWdfw
-----END CERTIFICATE-----

Consul service mesh uses gossip encryption to communicate with HCP Vault in order to securely transport TLS leaf certificates. The gossip encryption key is generated by HCP Consul while your terraform code was deploying, and is pre-loaded into your environment shell.

Retrieve the value from the environment.

$ export GOSSIP_KEY=$(echo $CONSUL_CONFIG | base64 --decode | jq -r '.encrypt') && echo $GOSSIP_KEY

$ export GOSSIP_KEY=$(echo $CONSUL_CONFIG | base64 --decode | jq -r '.encrypt') && echo $GOSSIP_KEY

Store the secret in vault kv

$ vault kv put -namespace=admin kv/gossipEncryptionKey key="${GOSSIP_KEY}"

$ vault kv put -namespace=admin kv/gossipEncryptionKey key="${GOSSIP_KEY}"

Validate the secret is written to HCP Vault.

$ vault kv get -namespace=admin kv/gossipEncryptionKey

  ======= Metadata =======
  Key                Value
  ---                -----
  created_time       2022-02-25T18:51:28.323739086Z
  custom_metadata    <nil>
  deletion_time      n/a
  destroyed          false
  version            1

  === Data ===
  Key    Value
  ---    -----
  key    pyEpgLmlnUtmGXh6ZB2ogw==


$ vault kv get -namespace=admin kv/gossipEncryptionKey

  ======= Metadata =======
  Key                Value
  ---                -----
  created_time       2022-02-25T18:51:28.323739086Z
  custom_metadata    <nil>
  deletion_time      n/a
  destroyed          false
  version            1

  === Data ===
  Key    Value
  ---    -----
  key    pyEpgLmlnUtmGXh6ZB2ogw==

Create an HCP Vault policy that gives the Consul service account read access to the gossip encryption key stored in Vault kv, and the CA certificate of the service account JWT. Attach the policy to a new role with the same name as the consul service account. This role uses the connect-ca policy you created earlier in this tutorial.

Create the policy file.

$ touch kubernetes-policy.hcl

$ touch kubernetes-policy.hcl

Add the following policy configuration to the policy file.

  path "admin/kv/gossipEncryption" {
  capabilities = ["read"]
  }

  path "admin/kv/caCertificate" {
  capabilities = ["read"]
  }
kubernetes-policy.hcl
Policy configuration
  path "admin/kv/gossipEncryption" {
  capabilities = ["read"]
  }

  path "admin/kv/caCertificate" {
  capabilities = ["read"]
  }

Write this policy to HCP Vault.

$ vault policy write -namespace=admin kubernetes-policy kubernetes-policy.hcl

  Success! Uploaded policy: kubernetes-policy
Write the policy to HCP Vault.
$ vault policy write -namespace=admin kubernetes-policy kubernetes-policy.hcl

  Success! Uploaded policy: kubernetes-policy

Attach the policy to a new role with the same name as the consul service account. This role uses the connect-ca policy you created earlier in this tutorial.

$ vault write -namespace=admin auth/kubernetes/role/$CONSUL_SERVICE_ACCOUNT \
    bound_service_account_names=$CONSUL_SERVICE_ACCOUNT \
    bound_service_account_namespaces=default \
    policies=kubernetes-policy,connect-ca
Write role to HCP Vault
$ vault write -namespace=admin auth/kubernetes/role/$CONSUL_SERVICE_ACCOUNT \
    bound_service_account_names=$CONSUL_SERVICE_ACCOUNT \
    bound_service_account_namespaces=default \
    policies=kubernetes-policy,connect-ca

»Deploy Consul Client Cluster to Amazon EKS

HCP Consul service mesh CA is using HCP Vault. You will next configure a Consul client cluster that registers to HCP Consul. This client cluster issues leaf certificates signed by the root certificate and assigns them to services inside the Consul service mesh.

If you deploy the Consul client cluster with Consul service mesh enabled, but do not configure the Consul Connect CA provider, installation will fail. Client clusters that register with Consul Connect server cluster in the same datacenter need to present the root certificate managed by HCP Consul. If you do not configure your client cluster, it will self-sign its own certificates, and present that certificate to HCP Consul, which fails. The role and policies you created in HCP Vault permit the Consul client cluster to retrieve and present the root certificate to HCP Consul.

Before installing the Consul client cluster, prepare the Kubernetes cluster with secrets generated by HCP Consul. The Consul client cluster accesses these values during installation when registering to the HCP Consul cluster. These values are preloaded into your shell as environment variables.

Seed the Kubernetes cluster with the HCP Consul CA certificate, the HCP Consul bootstrap token, the gossip encryption key, and your Consul Enterprise license. These items are stored as Kubernetes secrets.

Store the Consul CA certificate. (Note: This is separate from the Consul Connect CA Certificate)

$ export CONSUL_CA_DECODED=$(echo $CONSUL_CA | base64 --decode) && kubectl create secret generic "consul-ca-cert" --from-literal="tls.crt=${CONSUL_CA_DECODED}"

secret/consul-ca-cert created

$ export CONSUL_CA_DECODED=$(echo $CONSUL_CA | base64 --decode) && kubectl create secret generic "consul-ca-cert" --from-literal="tls.crt=${CONSUL_CA_DECODED}"

secret/consul-ca-cert created

Store the HCP Consul bootstrap token.

$ kubectl create secret generic "consul-bootstrap-token" --from-literal="token=${CONSUL_HTTP_TOKEN}"

secret/consul-bootstrap-token created

$ kubectl create secret generic "consul-bootstrap-token" --from-literal="token=${CONSUL_HTTP_TOKEN}"

secret/consul-bootstrap-token created

Store the Consul gossip encryption key

$ kubectl create secret generic "consul-gossip-key" --from-literal="key=${GOSSIP_KEY}"

secret/consul-gossip-key created

$ kubectl create secret generic "consul-gossip-key" --from-literal="key=${GOSSIP_KEY}"

secret/consul-gossip-key created

Store the Consul enterprise license

$ export CONSUL_ENT_LICENSE=$(cat /tmp/consul.hclic) && kubectl create secret generic "consul-ent-license" --from-literal="key=${CONSUL_ENT_LICENSE}"


secret/consul-ent-license created

$ export CONSUL_ENT_LICENSE=$(cat /tmp/consul.hclic) && kubectl create secret generic "consul-ent-license" --from-literal="key=${CONSUL_ENT_LICENSE}"


secret/consul-ent-license created

Create a Helm chart file for installing the Consul client cluster.

$ touch consul-values-client.yaml

$ touch consul-values-client.yaml

The helm chart for Consul on Kubernetes includes a global.secretsBackend stanza to specify HCP Vault as the Consul client's Connect CA backend. You will provide HCP Vault's information including the Root and Intermediate PKI paths created in HCP Vault, with the roles created in HCP Vault for Consul, you created in an earlier section.

Edit the file with the following configuration, replacing $CONSUL_HTTP_ADDR, $KUBE_CONTROL_PLANE, $VAULT_ADDR with their outputs. Remove https:// from the output of $CONSUL_HTTP_ADDR. Example output is shown below.

$ export HCP_CONSUL_ADDR=$(echo $CONSUL_HTTP_ADDR | sed 's/https:\/\///g')

$ export HCP_CONSUL_ADDR=$(echo $CONSUL_HTTP_ADDR | sed 's/https:\/\///g')

$ echo $HCP_CONSUL_ADDR && \
echo $KUBE_CONTROL_PLANE && \
echo $VAULT_ADDR

$ echo $HCP_CONSUL_ADDR && \
echo $KUBE_CONTROL_PLANE && \
echo $VAULT_ADDR

The above command prints similar output to the screen as shown in example output below.

dc1.private.consul.223350f5-f0b9-4d89-959d-aa8eab3184e5.aws.hashicorp.cloud
https://1A29E58888B55BE0B687C8DFA0F0E88A.gr7.us-east-1.eks.amazonaws.com:443
https://vault-cluster.private.vault.223350f5-f0b9-4d89-959d-aa8eab3184e5.aws.hashicorp.cloud:8200

dc1.private.consul.223350f5-f0b9-4d89-959d-aa8eab3184e5.aws.hashicorp.cloud
https://1A29E58888B55BE0B687C8DFA0F0E88A.gr7.us-east-1.eks.amazonaws.com:443
https://vault-cluster.private.vault.223350f5-f0b9-4d89-959d-aa8eab3184e5.aws.hashicorp.cloud:8200

global:
  name: consul
  enabled: false
  datacenter: dc1
  image: hashicorp/consul-enterprise:1.11.4-ent
  enableConsulNamespaces: true
  enterpriseLicense:
    secretName: consul-ent-license
    secretKey: key
  acls:
    manageSystemACLs: true
    bootstrapToken:
      secretName: consul-bootstrap-token
      secretKey: token
  gossipEncryption:
    secretName: consul-gossip-key
    secretKey: key
  tls:
    enabled: true
    enableAutoEncrypt: true
    caCert:
      secretName: consul-ca-cert
      secretKey: tls.crt
externalServers:
  enabled: true
  hosts:
  - $HCP_CONSUL_ADDR
  httpsPort: 443
  useSystemRoots: true
  k8sAuthMethodHost: $KUBE_CONTROL_PLANE
client:
  enabled: true
  grpc: true
  join:
  - $HCP_CONSUL_ADDR
connectInject:
  enabled: true
  envoyExtraArgs: "--component-log-level upstream:debug,http:debug,router:debug,config:debug"
  aclBindingRuleSelector: ''
  consulNamespaces:
    mirroringK8S: true
  transparentProxy:
    defaultEnabled: false
controller:
  enabled: true
secretsBackend:
  vault:
    enabled: true
    # https://www.consul.io/docs/k8s/helm#v-global-secretsbackend-vault-consulclientrole
    consulClientRole: consul-consul-client
    consulCARole: consul-consul-client
    # https://www.consul.io/docs/k8s/helm#v-global-secretsbackend-vault-connectca
    connectCA:
      address: $VAULT_ADDR
      rootPKIPath: /connect_root
      intermediatePKIPath: /connect_inter
      additionalConfig: |
        {
          "connect": [{
            "ca_config": [{
              "leaf_cert_ttl": "72h",
              "intermediate_cert_ttl": "8760h",
              "rotation_period": "2160h",
              "namespace": "admin"
            }]
          }]
        }
consul-values-client.yaml
1 2 3 4 5 6 7 8 9 1011121314151617181920212223242526272829303132333435363738394041424344454647484950515253545556575859606162636465666768global:
  name: consul
  enabled: false
  datacenter: dc1
  image: hashicorp/consul-enterprise:1.11.4-ent
  enableConsulNamespaces: true
  enterpriseLicense:
    secretName: consul-ent-license
    secretKey: key
  acls:
    manageSystemACLs: true
    bootstrapToken:
      secretName: consul-bootstrap-token
      secretKey: token
  gossipEncryption:
    secretName: consul-gossip-key
    secretKey: key
  tls:
    enabled: true
    enableAutoEncrypt: true
    caCert:
      secretName: consul-ca-cert
      secretKey: tls.crt
externalServers:
  enabled: true
  hosts:
  - $HCP_CONSUL_ADDR
  httpsPort: 443
  useSystemRoots: true
  k8sAuthMethodHost: $KUBE_CONTROL_PLANE
client:
  enabled: true
  grpc: true
  join:
  - $HCP_CONSUL_ADDR
connectInject:
  enabled: true
  envoyExtraArgs: "--component-log-level upstream:debug,http:debug,router:debug,config:debug"
  aclBindingRuleSelector: ''
  consulNamespaces:
    mirroringK8S: true
  transparentProxy:
    defaultEnabled: false
controller:
  enabled: true
secretsBackend:
  vault:
    enabled: true
    # https://www.consul.io/docs/k8s/helm#v-global-secretsbackend-vault-consulclientrole
    consulClientRole: consul-consul-client
    consulCARole: consul-consul-client
    # https://www.consul.io/docs/k8s/helm#v-global-secretsbackend-vault-connectca
    connectCA:
      address: $VAULT_ADDR
      rootPKIPath: /connect_root
      intermediatePKIPath: /connect_inter
      additionalConfig: |
        {
          "connect": [{
            "ca_config": [{
              "leaf_cert_ttl": "72h",
              "intermediate_cert_ttl": "8760h",
              "rotation_period": "2160h",
              "namespace": "admin"
            }]
          }]
        }

The workbench includes the Consul K8s CLI. The Consul K8s CLI is an alternative to installing Consul on Kubernetes instead of using Helm, Kubectl, and the Consul CLI.

Deploy Consul with the Consul K8s CLI. Press 'y' to accept and proceed with the installation.

$ consul-k8s install -namespace=default -config-file=./consul-values-client.yaml -timeout=3m

==> Pre-Install Checks
No existing installations found.
✓ No previous persistent volume claims found
✓ No previous secrets found

==> Consul Installation Summary

##...

Proceed with installation? (y/N) y

==> Running Installation
✓ Downloaded charts
--> creating 1 resource(s)
--> creating 44 resource(s)
--> beginning wait for 44 resources with timeout of 10m0s


$ consul-k8s install -namespace=default -config-file=./consul-values-client.yaml -timeout=3m

==> Pre-Install Checks
No existing installations found.
✓ No previous persistent volume claims found
✓ No previous secrets found

==> Consul Installation Summary

##...

Proceed with installation? (y/N) y

==> Running Installation
✓ Downloaded charts
--> creating 1 resource(s)
--> creating 44 resource(s)
--> beginning wait for 44 resources with timeout of 10m0s

»Verify Client Cluster Consul Connect Root Certificate

When the installation finishes, the HCP Consul Cluster has accepted the Amazon EKS Cluster presenting the Connect CA root certificate from HCP Vault for identity verification. To observe the Connect CA provider origin on the Consul client cluster, use the following command to query the Consul CLI.

Use the Consul Connect CLI to output the Consul Connect CA provider for the client cluster. Use the consul connect ca get-config command on any of the consul pods deployed to EKS.

$ export CLIENT_POD=$(kubectl get pods -l app=consul -l component=client -o json | jq -r '.items[0].metadata.name') && \
  kubectl exec -it \
  $CLIENT_POD -c consul -- consul connect ca get-config -token "${CONSUL_HTTP_TOKEN}"

$ export CLIENT_POD=$(kubectl get pods -l app=consul -l component=client -o json | jq -r '.items[0].metadata.name') && \
  kubectl exec -it \
  $CLIENT_POD -c consul -- consul connect ca get-config -token "${CONSUL_HTTP_TOKEN}"

{
  "Provider": "vault",
  "Config": {
          "Address": "https://vault-cluster.private.vault.223350f5-f0b9-4d89-959d-aa8eab3184e5.aws.hashicorp.cloud:8200",
          "IntermediateCertTTL": "8760h",
          "IntermediatePKIPath": "connect_inter",
          "LeafCertTTL": "72h",
          "Namespace": "admin",
          "PrivateKeyBits": 2048,
          "PrivateKeyType": "rsa",
          "RootPKIPath": "connect_root",
          "RotationPeriod": "2160h",
          "Token": "s.J4BoYPO4zjdkbXA8zhLtx9S4.XHw9L"
  },
  "State": null,
  "ForceWithoutCrossSigning": false,
  "CreateIndex": 9,
  "ModifyIndex": 883
}

{
  "Provider": "vault",
  "Config": {
          "Address": "https://vault-cluster.private.vault.223350f5-f0b9-4d89-959d-aa8eab3184e5.aws.hashicorp.cloud:8200",
          "IntermediateCertTTL": "8760h",
          "IntermediatePKIPath": "connect_inter",
          "LeafCertTTL": "72h",
          "Namespace": "admin",
          "PrivateKeyBits": 2048,
          "PrivateKeyType": "rsa",
          "RootPKIPath": "connect_root",
          "RotationPeriod": "2160h",
          "Token": "s.J4BoYPO4zjdkbXA8zhLtx9S4.XHw9L"
  },
  "State": null,
  "ForceWithoutCrossSigning": false,
  "CreateIndex": 9,
  "ModifyIndex": 883
}

»Deploy HashiCups with leaf certificates

With the root Connect CA now in place for the Consul client cluster, deploy HashiCups to the cluster. After, you will verify the generated leaf certificates, and the signing authority of those leaf certificates.

The HashiCups plan files are available in your workbench, located in the /tmp/hashicups/ directory.

Since ACLs are enabled, first apply Consul service intentions for the HashiCups services to have the ability to communicate with each other.

$ kubectl apply --filename /tmp/hashicups/crd-intentions.yaml

$ kubectl apply --filename /tmp/hashicups/crd-intentions.yaml

Deploy the HashiCups services to Kubernetes.

$ kubectl apply --filename /tmp/hashicups/postgres.yaml \
                --filename /tmp/hashicups/product-api.yaml \
                --filename /tmp/hashicups/public-api.yaml  \
                --filename /tmp/hashicups/payments.yaml  \
                --filename /tmp/hashicups/frontend.yaml

$ kubectl apply --filename /tmp/hashicups/postgres.yaml \
                --filename /tmp/hashicups/product-api.yaml \
                --filename /tmp/hashicups/public-api.yaml  \
                --filename /tmp/hashicups/payments.yaml  \
                --filename /tmp/hashicups/frontend.yaml

Confirm the HashiCups services are registered with Consul.

$ consul catalog services

consul
frontend
frontend-sidecar-proxy
payments
payments-sidecar-proxy
postgres
postgres-sidecar-proxy
products-api
products-api-sidecar-proxy
public-api
public-api-sidecar-proxy

$ consul catalog services

consul
frontend
frontend-sidecar-proxy
payments
payments-sidecar-proxy
postgres
postgres-sidecar-proxy
products-api
products-api-sidecar-proxy
public-api
public-api-sidecar-proxy

Confirm the HashiCups pods are in a working state, with kubectl. This can take several moments as the services scale and transfer into a "running" state.

$ kubectl get pods

NAME                                           READY   STATUS    RESTARTS   AGE
consul-client-4s6c6                            1/1     Running   0          13m
consul-client-gq7nn                            1/1     Running   0          13m
consul-client-qcqvf                            1/1     Running   0          13m
consul-connect-injector-596db4b65f-d8l4c       1/1     Running   0          13m
consul-connect-injector-596db4b65f-fsdxh       1/1     Running   0          13m
consul-controller-6b74cc589f-k848n             1/1     Running   0          13m
consul-webhook-cert-manager-8595bff784-fqqkt   1/1     Running   0          13m
frontend-57cfbf799-p7rth                       2/2     Running   0          4m32s
payments-5f9b7f86bc-88tn8                      2/2     Running   0          4m32s
postgres-b88484dd8-2gc97                       2/2     Running   0          4m32s
products-api-6b668b9bf9-6vcmt                  2/2     Running   0          4m32s
public-api-fbfdfd79d-f2fkr                     3/3     Running   0          4m32s
tutorial-76f6685fd6-pvjww                      1/1     Running   0          41m
vault-agent-injector-55555d9697-8xc9j          1/1     Running   0          31m


$ kubectl get pods

NAME                                           READY   STATUS    RESTARTS   AGE
consul-client-4s6c6                            1/1     Running   0          13m
consul-client-gq7nn                            1/1     Running   0          13m
consul-client-qcqvf                            1/1     Running   0          13m
consul-connect-injector-596db4b65f-d8l4c       1/1     Running   0          13m
consul-connect-injector-596db4b65f-fsdxh       1/1     Running   0          13m
consul-controller-6b74cc589f-k848n             1/1     Running   0          13m
consul-webhook-cert-manager-8595bff784-fqqkt   1/1     Running   0          13m
frontend-57cfbf799-p7rth                       2/2     Running   0          4m32s
payments-5f9b7f86bc-88tn8                      2/2     Running   0          4m32s
postgres-b88484dd8-2gc97                       2/2     Running   0          4m32s
products-api-6b668b9bf9-6vcmt                  2/2     Running   0          4m32s
public-api-fbfdfd79d-f2fkr                     3/3     Running   0          4m32s
tutorial-76f6685fd6-pvjww                      1/1     Running   0          41m
vault-agent-injector-55555d9697-8xc9j          1/1     Running   0          31m

»Validate leaf certificates for HashiCups services

With the HashiCups services deployed, verify that the leaf certificates are generated for the HashiCups services. You will use the postgres service to confirm. Use the /v1/agent/connect/ca/leaf Consul HTTP API endpoint to retrieve the leaf certificate bundle for this service inside the service mesh. Append the name of the service to the end of the API path. In this setup, use the /v1/agent/connect/ca/leaf/postgres endpoint to retrieve the certificate bundle.

NOTE: To learn more about certificate chains, bundles and their structure, read the Wikipedia article on the X.509 standard.

$ curl \
  --header "X-Consul-Token: $CONSUL_HTTP_TOKEN" \
  $CONSUL_HTTP_ADDR/v1/agent/connect/ca/leaf/postgres | \
  jq -r '. | "\(.CertPEM)"' > postgres-chain.crt

$ curl \
  --header "X-Consul-Token: $CONSUL_HTTP_TOKEN" \
  $CONSUL_HTTP_ADDR/v1/agent/connect/ca/leaf/postgres | \
  jq -r '. | "\(.CertPEM)"' > postgres-chain.crt

Observe the output of the saved file. (Edited to save space)

$ cat postgres-chain.crt

-----BEGIN CERTIFICATE-----
-----END CERTIFICATE-----

-----BEGIN CERTIFICATE-----
-----END CERTIFICATE-----

-----BEGIN CERTIFICATE-----
-----END CERTIFICATE-----


$ cat postgres-chain.crt

-----BEGIN CERTIFICATE-----
-----END CERTIFICATE-----

-----BEGIN CERTIFICATE-----
-----END CERTIFICATE-----

-----BEGIN CERTIFICATE-----
-----END CERTIFICATE-----

This bundle includes the certificate signing authority for the service, the intermediate CA certificate in the bundle, and certficate for the authority which signed the intermediate certificate. This bundle ensures the sender and Certificate Authority's trustworthiness. The first, or top certificate in the chain represents the postgres service in the chain.

Copy the top, or first certificate into its own file, starting from the BEGIN CERTIFICATE line at the top, to the first END CERTIFICATE line.

$ touch postgres.crt

$ touch postgres.crt

-----BEGIN CERTIFICATE-----
MIICyDCCAbCgAwIBAgIUe6PvDhGBf5mwXKy7eFF0dsjI6h8wDQYJKoZIhvcNAQEL
BQAwLzEtMCsGA1UEAxMkcHJpLTRyNmN2cDAudmF1bHQuY2EuZmU2NzdiNDQuY29u
c3VsMB4XDTIyMDMxNTAyMzU1NloXDTIyMDMxODAyMzYyNlowADBZMBMGByqGSM49
AgEGCCqGSM49AwEHA0IABM9G3DIcGzONqb8S/jkj1JEmUbupAkXKwu7JVOuLnSb7
5TlPEofjMNHY09trw10thsymG05mWkQhGNAEuTESPBajgdUwgdIwDgYDVR0PAQH/
BAQDAgOoMB0GA1UdJQQWMBQGCCsGAQUFBwMBBggrBgEFBQcDAjAdBgNVHQ4EFgQU
ZSqFPsCVYxW6ZI1kEyL2N7RWZ3YwHwYDVR0jBBgwFoAUCCUjXt+bXy8e8gqevJpd
H7UIxvcwYQYDVR0RAQH/BFcwVYZTc3BpZmZlOi8vZmU2NzdiNDQtNWUyYy05NTI4
LWU2OGEtMmQxNmY1ZmY4NjgyLmNvbnN1bC9ucy9kZWZhdWx0L2RjL2RjMS9zdmMv
cG9zdGdyZXMwDQYJKoZIhvcNAQELBQADggEBAJP6FO60eOk/iU+7RdlOhuF5PGvP
3Sxr70hRo51Ed0SW3rcJ5bzvZI6djmkdMhgcrQ/o+tFV/t/Yt2tSX5IFu+lauikJ
O+CCqZSiJccq+gU2VizbotAtFWT+ZEDWWMmrdqncrTADW2MaNpz1he8IoOOVkHKC
lROdQXSOzdy0DOxjzAS6EB9I9mx5JHWYfa6u+LucgkLjBn5SEuR+OYLdAKfUEblY
g1bRVr0zmNjleKMhINASgAe8cLArszpkF6TILtZh4P/tpE9OzugK4aAYT+qWRTTC
HOvwU4TqdjzoPdbTYRhlsegjpeUUIoSMCcTr34SAI/WRE0upr/h3F0E/2N0=
-----END CERTIFICATE-----

-----BEGIN CERTIFICATE-----
MIICyDCCAbCgAwIBAgIUe6PvDhGBf5mwXKy7eFF0dsjI6h8wDQYJKoZIhvcNAQEL
BQAwLzEtMCsGA1UEAxMkcHJpLTRyNmN2cDAudmF1bHQuY2EuZmU2NzdiNDQuY29u
c3VsMB4XDTIyMDMxNTAyMzU1NloXDTIyMDMxODAyMzYyNlowADBZMBMGByqGSM49
AgEGCCqGSM49AwEHA0IABM9G3DIcGzONqb8S/jkj1JEmUbupAkXKwu7JVOuLnSb7
5TlPEofjMNHY09trw10thsymG05mWkQhGNAEuTESPBajgdUwgdIwDgYDVR0PAQH/
BAQDAgOoMB0GA1UdJQQWMBQGCCsGAQUFBwMBBggrBgEFBQcDAjAdBgNVHQ4EFgQU
ZSqFPsCVYxW6ZI1kEyL2N7RWZ3YwHwYDVR0jBBgwFoAUCCUjXt+bXy8e8gqevJpd
H7UIxvcwYQYDVR0RAQH/BFcwVYZTc3BpZmZlOi8vZmU2NzdiNDQtNWUyYy05NTI4
LWU2OGEtMmQxNmY1ZmY4NjgyLmNvbnN1bC9ucy9kZWZhdWx0L2RjL2RjMS9zdmMv
cG9zdGdyZXMwDQYJKoZIhvcNAQELBQADggEBAJP6FO60eOk/iU+7RdlOhuF5PGvP
3Sxr70hRo51Ed0SW3rcJ5bzvZI6djmkdMhgcrQ/o+tFV/t/Yt2tSX5IFu+lauikJ
O+CCqZSiJccq+gU2VizbotAtFWT+ZEDWWMmrdqncrTADW2MaNpz1he8IoOOVkHKC
lROdQXSOzdy0DOxjzAS6EB9I9mx5JHWYfa6u+LucgkLjBn5SEuR+OYLdAKfUEblY
g1bRVr0zmNjleKMhINASgAe8cLArszpkF6TILtZh4P/tpE9OzugK4aAYT+qWRTTC
HOvwU4TqdjzoPdbTYRhlsegjpeUUIoSMCcTr34SAI/WRE0upr/h3F0E/2N0=
-----END CERTIFICATE-----

Inspect the certificate with the openssl tool, specifically observing the Signature Algorithm and X509v3 extensions sections of the command output.

$ openssl x509 -in postgres.crt -text

  Certificate:
    Data:
      Version: 3 (0x2)
      Serial Number:
        7b:a3:ef:0e:11:81:7f:99:b0:5c:ac:bb:78:51:74:76:c8:c8:ea:1f
    Signature Algorithm: sha256WithRSAEncryption
      Issuer: CN=pri-4r6cvp0.vault.ca.fe677b44.consul

. . .

      X509v3 extensions:
        X509v3 Key Usage: critical
          Digital Signature, Key Encipherment, Key Agreement
        X509v3 Extended Key Usage:
          TLS Web Server Authentication, TLS Web Client Authentication
        X509v3 Subject Key Identifier:
          65:2A:85:3E:C0:95:63:15:BA:64:8D:64:13:22:F6:37:B4:56:67:76
        X509v3 Authority Key Identifier:
          keyid:08:25:23:5E:DF:9B:5F:2F:1E:F2:0A:9E:BC:9A:5D:1F:B5:08:C6:F7

$ openssl x509 -in postgres.crt -text

  Certificate:
    Data:
      Version: 3 (0x2)
      Serial Number:
        7b:a3:ef:0e:11:81:7f:99:b0:5c:ac:bb:78:51:74:76:c8:c8:ea:1f
    Signature Algorithm: sha256WithRSAEncryption
      Issuer: CN=pri-4r6cvp0.vault.ca.fe677b44.consul

. . .

      X509v3 extensions:
        X509v3 Key Usage: critical
          Digital Signature, Key Encipherment, Key Agreement
        X509v3 Extended Key Usage:
          TLS Web Server Authentication, TLS Web Client Authentication
        X509v3 Subject Key Identifier:
          65:2A:85:3E:C0:95:63:15:BA:64:8D:64:13:22:F6:37:B4:56:67:76
        X509v3 Authority Key Identifier:
          keyid:08:25:23:5E:DF:9B:5F:2F:1E:F2:0A:9E:BC:9A:5D:1F:B5:08:C6:F7

By referencing the Standard Extensions in the X509 extensions configuration guide, you can confirm that this service certificate is not a certificate assigned to a Certificate Authority, via the absence of the "Basic Extensions" section in this output. This means the certificate was issued and signed by a CA. Verify the origin of this certificate by observing the value of X509v3 Authority Key Identifier and save it as a shell variable.

$ export POSTGRES_CERT_SERIAL=$(echo "08:25:23:5E:DF:9B:5F:2F:1E:F2:0A:9E:BC:9A:5D:1F:B5:08:C6:F7")

$ export POSTGRES_CERT_SERIAL=$(echo "08:25:23:5E:DF:9B:5F:2F:1E:F2:0A:9E:BC:9A:5D:1F:B5:08:C6:F7")

This serial number represents the X509v3 Subject Key Identifier for the next certificate in the bundle. This serial number is not available to query in HCP Vault directly. You will retrieve this serial number indirectly. First, return to the certificate bundle you saved at the beginning, and copy the contents of the two remaining certificates into separate files. Inspect each file's X509v3 Subject Key Identifier value.

$ echo postgres-chain.crt && touch postgres-second.crt

. . .

-----BEGIN CERTIFICATE-----
MIIC9DCCApqgAwIBAgIBDDAKBggqhkjOPQQDAjAxMS8wLQYDVQQDEyZwcmkteXQ0
dDdxMGcuY29uc3VsLmNhLmZlNjc3YjQ0LmNvbnN1bDAeFw0yMjAzMTUwMjI5MTJa
Fw0yMjAzMjIwMjI5MTJaMDAxLjAsBgNVBAMTJXByaS0xaWtxbGxlYy52YXVsdC5j
YS5mZTY3N2I0NC5jb25zdWwwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIB
AQCwRkJts2hb2cm9QYMsgoUCKDr/aaLnYYtvJW4ZTT8dlKeNJV3YUHsUjAz057Dc
UEQSycptYQe0UV3wGM7TcS2QPdm2PiFJ9i23RGpB62S30oPo91gWMF9o5SGvq1E/
yLeljv7Ner+e6tJdX1Dq8tvt1+DHkz6+5S+mu3bRYNmIPPK1WFLgiEMfWP2iSY6F
FB4NYHEtbesYeChHExai88GGeQc7XNG1+KTW9c6ac8+kXe8g4ubCb7MwMOfWh6y2
nfUeXsggSjxTN0zCtwQtsEI6+iHb7UUsIVL060s1zsdyFqMRVL8mxIuDRHSFB87+
GsZrNxONozbajACi+uSOo785AgMBAAGjgdgwgdUwDgYDVR0PAQH/BAQDAgEGMA8G
A1UdEwEB/wQFMAMBAf8wHQYDVR0OBBYEFGvW6s3TmcBs5Jm0XlVIgPBKBLFpMCsG
A1UdIwQkMCKAIB00ikqiDzjdmKNkYJzk2YBTGn9KaI78O/n6guQNVYQzMGYGA1Ud
EQRfMF2CJXByaS0xaWtxbGxlYy52YXVsdC5jYS5mZTY3N2I0NC5jb25zdWyGNHNw
aWZmZTovL2ZlNjc3YjQ0LTVlMmMtOTUyOC1lNjhhLTJkMTZmNWZmODY4Mi5jb25z
dWwwCgYIKoZIzj0EAwIDSAAwRQIhAP0Im485aMNTdF3vlWCFgIZBe++Qe3J+TB7+
c+BvBktCAiAZ80/rPlwboxt1QM6izc2fDlTg8fFAqX3lYhD5rqd8Uw==
-----END CERTIFICATE-----

. . .


$ echo postgres-chain.crt && touch postgres-second.crt

. . .

-----BEGIN CERTIFICATE-----
MIIC9DCCApqgAwIBAgIBDDAKBggqhkjOPQQDAjAxMS8wLQYDVQQDEyZwcmkteXQ0
dDdxMGcuY29uc3VsLmNhLmZlNjc3YjQ0LmNvbnN1bDAeFw0yMjAzMTUwMjI5MTJa
Fw0yMjAzMjIwMjI5MTJaMDAxLjAsBgNVBAMTJXByaS0xaWtxbGxlYy52YXVsdC5j
YS5mZTY3N2I0NC5jb25zdWwwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIB
AQCwRkJts2hb2cm9QYMsgoUCKDr/aaLnYYtvJW4ZTT8dlKeNJV3YUHsUjAz057Dc
UEQSycptYQe0UV3wGM7TcS2QPdm2PiFJ9i23RGpB62S30oPo91gWMF9o5SGvq1E/
yLeljv7Ner+e6tJdX1Dq8tvt1+DHkz6+5S+mu3bRYNmIPPK1WFLgiEMfWP2iSY6F
FB4NYHEtbesYeChHExai88GGeQc7XNG1+KTW9c6ac8+kXe8g4ubCb7MwMOfWh6y2
nfUeXsggSjxTN0zCtwQtsEI6+iHb7UUsIVL060s1zsdyFqMRVL8mxIuDRHSFB87+
GsZrNxONozbajACi+uSOo785AgMBAAGjgdgwgdUwDgYDVR0PAQH/BAQDAgEGMA8G
A1UdEwEB/wQFMAMBAf8wHQYDVR0OBBYEFGvW6s3TmcBs5Jm0XlVIgPBKBLFpMCsG
A1UdIwQkMCKAIB00ikqiDzjdmKNkYJzk2YBTGn9KaI78O/n6guQNVYQzMGYGA1Ud
EQRfMF2CJXByaS0xaWtxbGxlYy52YXVsdC5jYS5mZTY3N2I0NC5jb25zdWyGNHNw
aWZmZTovL2ZlNjc3YjQ0LTVlMmMtOTUyOC1lNjhhLTJkMTZmNWZmODY4Mi5jb25z
dWwwCgYIKoZIzj0EAwIDSAAwRQIhAP0Im485aMNTdF3vlWCFgIZBe++Qe3J+TB7+
c+BvBktCAiAZ80/rPlwboxt1QM6izc2fDlTg8fFAqX3lYhD5rqd8Uw==
-----END CERTIFICATE-----

. . .


$ openssl x509 -in postgres-second.crt -text

  X509v3 extensions:
    X509v3 Key Usage: critical
      Certificate Sign, CRL Sign
    X509v3 Basic Constraints: critical
      CA:TRUE
    X509v3 Subject Key Identifier:
      6B:D6:EA:CD:D3:99:C0:6C:E4:99:B4:5E:55:48:80:F0:4A:04:B1:69
    X509v3 Authority Key Identifier:
      keyid:1D:34:8A:4A:A2:0F:38:DD:98:A3:64:60:9C:E4:D9:80:53:1A:7F:4A:68:8E:FC:3B:F9:FA:82:E4:0D:55:84:33

$ openssl x509 -in postgres-second.crt -text

  X509v3 extensions:
    X509v3 Key Usage: critical
      Certificate Sign, CRL Sign
    X509v3 Basic Constraints: critical
      CA:TRUE
    X509v3 Subject Key Identifier:
      6B:D6:EA:CD:D3:99:C0:6C:E4:99:B4:5E:55:48:80:F0:4A:04:B1:69
    X509v3 Authority Key Identifier:
      keyid:1D:34:8A:4A:A2:0F:38:DD:98:A3:64:60:9C:E4:D9:80:53:1A:7F:4A:68:8E:FC:3B:F9:FA:82:E4:0D:55:84:33

The middle certificate's Subject Key Identifier does not match the service certificate's Authority Key Identifier. Inspect the certificate at the bottom of the certificate bundle.

WARNING Depending on the order returned in the certificate bundle, the middle certificate may match the authority key identifier. If so, skip to the next step for the last certificate in the bundle to complete the task.

$ touch postgres-third.crt

$ touch postgres-third.crt

-----BEGIN CERTIFICATE-----
MIIDuTCCAqGgAwIBAgIUdTaUn6lNFL5ujLakSHxqaxNYjkswDQYJKoZIhvcNAQEL
BQAwMDEuMCwGA1UEAxMlcHJpLTFpa3FsbGVjLnZhdWx0LmNhLmZlNjc3YjQ0LmNv
bnN1bDAeFw0yMjAzMTUwMjI5NDNaFw0yMzAzMTUwMjMwMTNaMC8xLTArBgNVBAMT
JHByaS00cjZjdnAwLnZhdWx0LmNhLmZlNjc3YjQ0LmNvbnN1bDCCASIwDQYJKoZI
hvcNAQEBBQADggEPADCCAQoCggEBAMDgnl9GuNsHM43RhNbXIuJHfLkgR4d4Rr12
6znkcaKUbpK11nIcgSmf0hmgDqsQpKs6Oavs6a8WPcQD+sqjZgFuUIWAbUU6jmsj
Fy3DzCZctne7kHIGFpjqClm/tNIOpkwcZbcodmBZUTMW5Ga62xVsi7pSdZn/kxbu
gK6VmwCpZ6qDgxUQSWE7/uQclrpB+ReWI19vpzPrJCnLD+SdDnQdH08Qc32+3WtP
CUE9FEOQMQW5b18ATCt1yQiYQmMxySgZz3BjNG95Ed8jLSB3EElZWGVCeaWtaxRN
5OiFtyE4rh4mSxEsPa7RAvnSF52g3bafIiJEAA651hRlhDbysW8CAwEAAaOByzCB
yDAOBgNVHQ8BAf8EBAMCAQYwDwYDVR0TAQH/BAUwAwEB/zAdBgNVHQ4EFgQUCCUj
Xt+bXy8e8gqevJpdH7UIxvcwHwYDVR0jBBgwFoAUa9bqzdOZwGzkmbReVUiA8EoE
sWkwZQYDVR0RBF4wXIIkcHJpLTRyNmN2cDAudmF1bHQuY2EuZmU2NzdiNDQuY29u
c3VshjRzcGlmZmU6Ly9mZTY3N2I0NC01ZTJjLTk1MjgtZTY4YS0yZDE2ZjVmZjg2
ODIuY29uc3VsMA0GCSqGSIb3DQEBCwUAA4IBAQCKjUkbc1IA2V0Jf4LVsNEDu9Nc
hDTIGG/0V3cVrmSlJddZIdGhwRsA6ZY8oaKjzwS1ou/2jSkmBlsoLQq/CrywqJ+7
ZQNGDp8sSZ3S9Uy8VPVNTfHzmyKhW8NavU1Tc77Bnp2boAvWJob0FYhskEIWiVEM
J1y1d+ObtcakJAzJXnPKJnGD4VLKsL0lnG9YhumvRuoAecLwpOIZfxd8JfwtAaLO
m9foLoa0AviNy5084z1O8pvCOt3IsdZ9LaexuulIIlu+5cVqO6fyBcNVFrpmSFFG
MThcfl7BLFCRWsdzxPjLCKC4MSYkgWIDnX2je+Dkeh7f1c1WKXxKhqP+b+WO
-----END CERTIFICATE-----

-----BEGIN CERTIFICATE-----
MIIDuTCCAqGgAwIBAgIUdTaUn6lNFL5ujLakSHxqaxNYjkswDQYJKoZIhvcNAQEL
BQAwMDEuMCwGA1UEAxMlcHJpLTFpa3FsbGVjLnZhdWx0LmNhLmZlNjc3YjQ0LmNv
bnN1bDAeFw0yMjAzMTUwMjI5NDNaFw0yMzAzMTUwMjMwMTNaMC8xLTArBgNVBAMT
JHByaS00cjZjdnAwLnZhdWx0LmNhLmZlNjc3YjQ0LmNvbnN1bDCCASIwDQYJKoZI
hvcNAQEBBQADggEPADCCAQoCggEBAMDgnl9GuNsHM43RhNbXIuJHfLkgR4d4Rr12
6znkcaKUbpK11nIcgSmf0hmgDqsQpKs6Oavs6a8WPcQD+sqjZgFuUIWAbUU6jmsj
Fy3DzCZctne7kHIGFpjqClm/tNIOpkwcZbcodmBZUTMW5Ga62xVsi7pSdZn/kxbu
gK6VmwCpZ6qDgxUQSWE7/uQclrpB+ReWI19vpzPrJCnLD+SdDnQdH08Qc32+3WtP
CUE9FEOQMQW5b18ATCt1yQiYQmMxySgZz3BjNG95Ed8jLSB3EElZWGVCeaWtaxRN
5OiFtyE4rh4mSxEsPa7RAvnSF52g3bafIiJEAA651hRlhDbysW8CAwEAAaOByzCB
yDAOBgNVHQ8BAf8EBAMCAQYwDwYDVR0TAQH/BAUwAwEB/zAdBgNVHQ4EFgQUCCUj
Xt+bXy8e8gqevJpdH7UIxvcwHwYDVR0jBBgwFoAUa9bqzdOZwGzkmbReVUiA8EoE
sWkwZQYDVR0RBF4wXIIkcHJpLTRyNmN2cDAudmF1bHQuY2EuZmU2NzdiNDQuY29u
c3VshjRzcGlmZmU6Ly9mZTY3N2I0NC01ZTJjLTk1MjgtZTY4YS0yZDE2ZjVmZjg2
ODIuY29uc3VsMA0GCSqGSIb3DQEBCwUAA4IBAQCKjUkbc1IA2V0Jf4LVsNEDu9Nc
hDTIGG/0V3cVrmSlJddZIdGhwRsA6ZY8oaKjzwS1ou/2jSkmBlsoLQq/CrywqJ+7
ZQNGDp8sSZ3S9Uy8VPVNTfHzmyKhW8NavU1Tc77Bnp2boAvWJob0FYhskEIWiVEM
J1y1d+ObtcakJAzJXnPKJnGD4VLKsL0lnG9YhumvRuoAecLwpOIZfxd8JfwtAaLO
m9foLoa0AviNy5084z1O8pvCOt3IsdZ9LaexuulIIlu+5cVqO6fyBcNVFrpmSFFG
MThcfl7BLFCRWsdzxPjLCKC4MSYkgWIDnX2je+Dkeh7f1c1WKXxKhqP+b+WO
-----END CERTIFICATE-----

$ openssl x509 -in postgres-third.crt -text

Serial Number:
  75:36:94:9f:a9:4d:14:be:6e:8c:b6:a4:48:7c:6a:6b:13:58:8e:4b

X509v3 extensions:
  X509v3 Key Usage: critical
    Certificate Sign, CRL Sign
  X509v3 Basic Constraints: critical
    CA:TRUE
  X509v3 Subject Key Identifier:
    08:25:23:5E:DF:9B:5F:2F:1E:F2:0A:9E:BC:9A:5D:1F:B5:08:C6:F7
  X509v3 Authority Key Identifier:
    keyid:6B:D6:EA:CD:D3:99:C0:6C:E4:99:B4:5E:55:48:80:F0:4A:04:B1:69

$ openssl x509 -in postgres-third.crt -text

Serial Number:
  75:36:94:9f:a9:4d:14:be:6e:8c:b6:a4:48:7c:6a:6b:13:58:8e:4b

X509v3 extensions:
  X509v3 Key Usage: critical
    Certificate Sign, CRL Sign
  X509v3 Basic Constraints: critical
    CA:TRUE
  X509v3 Subject Key Identifier:
    08:25:23:5E:DF:9B:5F:2F:1E:F2:0A:9E:BC:9A:5D:1F:B5:08:C6:F7
  X509v3 Authority Key Identifier:
    keyid:6B:D6:EA:CD:D3:99:C0:6C:E4:99:B4:5E:55:48:80:F0:4A:04:B1:69

This certificate's Subject Key Identifier matches the Authority Key Identifier for the postgres service.

$ echo $POSTGRES_CERT_SERIAL && echo "08:25:23:5E:DF:9B:5F:2F:1E:F2:0A:9E:BC:9A:5D:1F:B5:08:C6:F7"

08:25:23:5E:DF:9B:5F:2F:1E:F2:0A:9E:BC:9A:5D:1F:B5:08:C6:F7
08:25:23:5E:DF:9B:5F:2F:1E:F2:0A:9E:BC:9A:5D:1F:B5:08:C6:F7

$ echo $POSTGRES_CERT_SERIAL && echo "08:25:23:5E:DF:9B:5F:2F:1E:F2:0A:9E:BC:9A:5D:1F:B5:08:C6:F7"

08:25:23:5E:DF:9B:5F:2F:1E:F2:0A:9E:BC:9A:5D:1F:B5:08:C6:F7
08:25:23:5E:DF:9B:5F:2F:1E:F2:0A:9E:BC:9A:5D:1F:B5:08:C6:F7

Query the connect_inter pki path for this certificate's Serial Number value, replacing the colons with slashes. The replacement allows the serial number to be included in as a search parameter for the Vault HTTP API.

$ export INTERMEDIATE_CERT_SERIAL=$(echo "75:36:94:9f:a9:4d:14:be:6e:8c:b6:a4:48:7c:6a:6b:13:58:8e:4b" | sed 's/:/-/'g)

$ export INTERMEDIATE_CERT_SERIAL=$(echo "75:36:94:9f:a9:4d:14:be:6e:8c:b6:a4:48:7c:6a:6b:13:58:8e:4b" | sed 's/:/-/'g)

Query the value of the Serial Number in the HCP Vault API.

$ curl \
 --header "X-Vault-Token: $VAULT_TOKEN" \
 --header "X-Vault-Namespace: $VAULT_NAMESPACE" \
 $VAULT_ADDR/v1/connect_inter/cert/$INTERMEDIATE_CERT_SERIAL | jq

$ curl \
 --header "X-Vault-Token: $VAULT_TOKEN" \
 --header "X-Vault-Namespace: $VAULT_NAMESPACE" \
 $VAULT_ADDR/v1/connect_inter/cert/$INTERMEDIATE_CERT_SERIAL | jq

  {
    "request_id": "1fcd5e09-6806-f516-2614-f09eefcc1bdf",
    "lease_id": "",
    "renewable": false,
    "lease_duration": 0,
    "data": {
    "certificate": "-----BEGIN CERTIFICATE-----\nMIIDuTCCAqGgAwIBAgIUdTaUn6lNFL5ujLakSHxqaxNYjkswDQYJKoZIhvcNAQEL\nBQAwMDEuMCwGA1UEAxMlcHJpLTFpa3FsbGVjLnZhdWx0LmNhLmZlNjc3YjQ0LmNv\nbnN1bDAeFw0yMjAzMTUwMjI5NDNaFw0yMzAzMTUwMjMwMTNaMC8xLTArBgNVBAMT\nJHByaS00cjZjdnAwLnZhdWx0LmNhLmZlNjc3YjQ0LmNvbnN1bDCCASIwDQYJKoZI\nhvcNAQEBBQADggEPADCCAQoCggEBAMDgnl9GuNsHM43RhNbXIuJHfLkgR4d4Rr12\n6znkcaKUbpK11nIcgSmf0hmgDqsQpKs6Oavs6a8WPcQD+sqjZgFuUIWAbUU6jmsj\nFy3DzCZctne7kHIGFpjqClm/tNIOpkwcZbcodmBZUTMW5Ga62xVsi7pSdZn/kxbu\ngK6VmwCpZ6qDgxUQSWE7/uQclrpB+ReWI19vpzPrJCnLD+SdDnQdH08Qc32+3WtP\nCUE9FEOQMQW5b18ATCt1yQiYQmMxySgZz3BjNG95Ed8jLSB3EElZWGVCeaWtaxRN\n5OiFtyE4rh4mSxEsPa7RAvnSF52g3bafIiJEAA651hRlhDbysW8CAwEAAaOByzCB\nyDAOBgNVHQ8BAf8EBAMCAQYwDwYDVR0TAQH/BAUwAwEB/zAdBgNVHQ4EFgQUCCUj\nXt+bXy8e8gqevJpdH7UIxvcwHwYDVR0jBBgwFoAUa9bqzdOZwGzkmbReVUiA8EoE\nsWkwZQYDVR0RBF4wXIIkcHJpLTRyNmN2cDAudmF1bHQuY2EuZmU2NzdiNDQuY29u\nc3VshjRzcGlmZmU6Ly9mZTY3N2I0NC01ZTJjLTk1MjgtZTY4YS0yZDE2ZjVmZjg2\nODIuY29uc3VsMA0GCSqGSIb3DQEBCwUAA4IBAQCKjUkbc1IA2V0Jf4LVsNEDu9Nc\nhDTIGG/0V3cVrmSlJddZIdGhwRsA6ZY8oaKjzwS1ou/2jSkmBlsoLQq/CrywqJ+7\nZQNGDp8sSZ3S9Uy8VPVNTfHzmyKhW8NavU1Tc77Bnp2boAvWJob0FYhskEIWiVEM\nJ1y1d+ObtcakJAzJXnPKJnGD4VLKsL0lnG9YhumvRuoAecLwpOIZfxd8JfwtAaLO\nm9foLoa0AviNy5084z1O8pvCOt3IsdZ9LaexuulIIlu+5cVqO6fyBcNVFrpmSFFG\nMThcfl7BLFCRWsdzxPjLCKC4MSYkgWIDnX2je+Dkeh7f1c1WKXxKhqP+b+WO\n-----END CERTIFICATE-----",
    "revocation_time": 0
  },
    "wrap_info": null,
    "warnings": null,
    "auth": null
  }

  {
    "request_id": "1fcd5e09-6806-f516-2614-f09eefcc1bdf",
    "lease_id": "",
    "renewable": false,
    "lease_duration": 0,
    "data": {
    "certificate": "-----BEGIN CERTIFICATE-----\nMIIDuTCCAqGgAwIBAgIUdTaUn6lNFL5ujLakSHxqaxNYjkswDQYJKoZIhvcNAQEL\nBQAwMDEuMCwGA1UEAxMlcHJpLTFpa3FsbGVjLnZhdWx0LmNhLmZlNjc3YjQ0LmNv\nbnN1bDAeFw0yMjAzMTUwMjI5NDNaFw0yMzAzMTUwMjMwMTNaMC8xLTArBgNVBAMT\nJHByaS00cjZjdnAwLnZhdWx0LmNhLmZlNjc3YjQ0LmNvbnN1bDCCASIwDQYJKoZI\nhvcNAQEBBQADggEPADCCAQoCggEBAMDgnl9GuNsHM43RhNbXIuJHfLkgR4d4Rr12\n6znkcaKUbpK11nIcgSmf0hmgDqsQpKs6Oavs6a8WPcQD+sqjZgFuUIWAbUU6jmsj\nFy3DzCZctne7kHIGFpjqClm/tNIOpkwcZbcodmBZUTMW5Ga62xVsi7pSdZn/kxbu\ngK6VmwCpZ6qDgxUQSWE7/uQclrpB+ReWI19vpzPrJCnLD+SdDnQdH08Qc32+3WtP\nCUE9FEOQMQW5b18ATCt1yQiYQmMxySgZz3BjNG95Ed8jLSB3EElZWGVCeaWtaxRN\n5OiFtyE4rh4mSxEsPa7RAvnSF52g3bafIiJEAA651hRlhDbysW8CAwEAAaOByzCB\nyDAOBgNVHQ8BAf8EBAMCAQYwDwYDVR0TAQH/BAUwAwEB/zAdBgNVHQ4EFgQUCCUj\nXt+bXy8e8gqevJpdH7UIxvcwHwYDVR0jBBgwFoAUa9bqzdOZwGzkmbReVUiA8EoE\nsWkwZQYDVR0RBF4wXIIkcHJpLTRyNmN2cDAudmF1bHQuY2EuZmU2NzdiNDQuY29u\nc3VshjRzcGlmZmU6Ly9mZTY3N2I0NC01ZTJjLTk1MjgtZTY4YS0yZDE2ZjVmZjg2\nODIuY29uc3VsMA0GCSqGSIb3DQEBCwUAA4IBAQCKjUkbc1IA2V0Jf4LVsNEDu9Nc\nhDTIGG/0V3cVrmSlJddZIdGhwRsA6ZY8oaKjzwS1ou/2jSkmBlsoLQq/CrywqJ+7\nZQNGDp8sSZ3S9Uy8VPVNTfHzmyKhW8NavU1Tc77Bnp2boAvWJob0FYhskEIWiVEM\nJ1y1d+ObtcakJAzJXnPKJnGD4VLKsL0lnG9YhumvRuoAecLwpOIZfxd8JfwtAaLO\nm9foLoa0AviNy5084z1O8pvCOt3IsdZ9LaexuulIIlu+5cVqO6fyBcNVFrpmSFFG\nMThcfl7BLFCRWsdzxPjLCKC4MSYkgWIDnX2je+Dkeh7f1c1WKXxKhqP+b+WO\n-----END CERTIFICATE-----",
    "revocation_time": 0
  },
    "wrap_info": null,
    "warnings": null,
    "auth": null
  }

This shell output confirms the certificate generated for the postgres service is signed by an authority that has its CA certificate stored in HCP Vault. In this setup, the signing authority is the Consul Connect service mesh Certificate Authority backed by HCP Vault, as previously configured.

As a challenge, trace the intermediate certificate to the root certificate by using these directions on the middle certificate listed in the tutorial, and final certificate in the certificate chain, comparing the values against the middle certificate in the chain.

»Validate HashiCups deployment status.

Exit the workbench. You will verify the HashiCups services are functioning as expected.

$ exit

$ exit

Update your workbench's kubeconfig file with your cluster information.

$ aws eks --region us-east-1 update-kubeconfig --name tutorialCluster

$ aws eks --region us-east-1 update-kubeconfig --name tutorialCluster

Verify the rest of the HashiCups pods are in a working state with kubectl.

$ kubectl get pods

  consul-client-vbf7l                            1/1     Running   0          3m45s
  consul-client-w2h72                            1/1     Running   0          3m45s
  consul-client-x99x5                            1/1     Running   0          3m45s
  consul-connect-injector-596db4b65f-6wcrx       1/1     Running   0          3m45s
  consul-connect-injector-596db4b65f-984rx       1/1     Running   0          3m45s
  consul-controller-6b74cc589f-frdlf             1/1     Running   0          3m45s
  consul-webhook-cert-manager-8595bff784-nlpfk   1/1     Running   0          3m45s
  frontend-5748489f8f-6g4cd                      2/2     Running   0          24s
  payments-5f9b7f86bc-pn2fp                      2/2     Running   0          32s
  postgres-b88484dd8-fj4bq                       2/2     Running   0          86s
  products-api-6b668b9bf9-5vvbn                  2/2     Running   0          47s
  public-api-fbfdfd79d-lls7f                     3/3     Running   0          39s
  tutorial-5fccc6b5f5-h4mcc                      1/1     Running   0          22m
  vault-agent-injector-55555d9697-cf7dg          1/1     Running   0          14m

$ kubectl get pods

  consul-client-vbf7l                            1/1     Running   0          3m45s
  consul-client-w2h72                            1/1     Running   0          3m45s
  consul-client-x99x5                            1/1     Running   0          3m45s
  consul-connect-injector-596db4b65f-6wcrx       1/1     Running   0          3m45s
  consul-connect-injector-596db4b65f-984rx       1/1     Running   0          3m45s
  consul-controller-6b74cc589f-frdlf             1/1     Running   0          3m45s
  consul-webhook-cert-manager-8595bff784-nlpfk   1/1     Running   0          3m45s
  frontend-5748489f8f-6g4cd                      2/2     Running   0          24s
  payments-5f9b7f86bc-pn2fp                      2/2     Running   0          32s
  postgres-b88484dd8-fj4bq                       2/2     Running   0          86s
  products-api-6b668b9bf9-5vvbn                  2/2     Running   0          47s
  public-api-fbfdfd79d-lls7f                     3/3     Running   0          39s
  tutorial-5fccc6b5f5-h4mcc                      1/1     Running   0          22m
  vault-agent-injector-55555d9697-cf7dg          1/1     Running   0          14m

Use kubectl to create a port forward for the frontend, and public-api pods to your local environment (Not the workbench). This example sets a shell variable for the name of the service, passing it to kubectl's port-forward command. This creates a connection from the local pod port, to your local environment.

$ export FRONTEND=$(kubectl get pods -l app=frontend -o json | jq -r '.items[0].metadata.name') && kubectl port-forward $FRONTEND 3500:3000

Forwarding from 127.0.0.1:3500 -> 3000
Forwarding from [::1]:3500 -> 3000
Handling connection for 3500

$ export FRONTEND=$(kubectl get pods -l app=frontend -o json | jq -r '.items[0].metadata.name') && kubectl port-forward $FRONTEND 3500:3000

Forwarding from 127.0.0.1:3500 -> 3000
Forwarding from [::1]:3500 -> 3000
Handling connection for 3500

$ export PUBLIC_API=$(kubectl get pods -l app=public-api -o json | jq -r '.items[0].metadata.name') && kubectl port-forward $PUBLIC_API 8080:8080

Forwarding from 127.0.0.1:8080 -> 8080
Forwarding from [::1]:3500 -> 3000
Handling connection for 3500

$ export PUBLIC_API=$(kubectl get pods -l app=public-api -o json | jq -r '.items[0].metadata.name') && kubectl port-forward $PUBLIC_API 8080:8080

Forwarding from 127.0.0.1:8080 -> 8080
Forwarding from [::1]:3500 -> 3000
Handling connection for 3500

Navigate to the frontend URL http://localhost:3500.

You can now observe the HashiCups application deployed to the client cluster, in your local browser.

Image of the HashiCups storefront, picturing 10 unique beverages with a pun on a HashiCorp product

»Leaf certificate expiration and verification

The /agent/connect/ca/leaf/$SERVICE REST API endpoint is used to return information about the leaf certificate generated for the Consul service.

Log back in to your workbench.

$ kubectl exec -it "${WORKBENCH}" /bin/bash

$ kubectl exec -it "${WORKBENCH}" /bin/bash

To set a leaf certificate's expiration, the leaf_cert_ttl option is set in the helm chart values file. This configuration is available to set in the free-text secretsBackend.vault.connectCA.additionalConfig stanza in consul-values-client.yaml. This setting applies to all leaf certificates signed by the Consul Connect CA. The minimum rotation period for leaf certificates is one hour. This process is managed automatically by HCP Consul and HCP Vault.

{
 "connect": [{
 "ca_config": [{
 "leaf_cert_ttl": "72h",
 "intermediate_cert_ttl": "8760h",
 "rotation_period": "2160h"
   ]}
  ]}
}
consul-values-client.yaml
{
 "connect": [{
 "ca_config": [{
 "leaf_cert_ttl": "72h",
 "intermediate_cert_ttl": "8760h",
 "rotation_period": "2160h"
   ]}
  ]}
}

Verify a certificate's TTL by inspecting the response when calling the /agent/connect/ca/leaf/$SERVICE endpoint, with curl.

$ curl --verbose --header "X-Consul-Token: $CONSUL_HTTP_TOKEN" $CONSUL_HTTP_ADDR/v1/agent/connect/ca/leaf/postgres | jq

$ curl --verbose --header "X-Consul-Token: $CONSUL_HTTP_TOKEN" $CONSUL_HTTP_ADDR/v1/agent/connect/ca/leaf/postgres | jq

< HTTP/2 200
< content-type: application/json
< date: Wed, 02 Mar 2022 03:27:45 GMT
< strict-transport-security: max-age=31536000
< vary: Accept-Encoding
< x-cache: MISS
< x-consul-default-acl-policy: deny
< x-consul-index: 1527

  {
  "SerialNumber": "2b:07:f4:f6:a5:3d:84:2e:fa:1a:27:8b:af:57:ba:1b:84:b1:1e:91",
  "CertPEM": "Edited for Brevity",
  "PrivateKeyPEM": "Edited for Brevity",
  "Service": "postgres",
  "ServiceURI": "spiffe://dc5c5a9f-706c-de1f-d10f-f3c0645c46bc.consul/ns/default/dc/dc1/svc/postgres",
  "ValidAfter": "2022-03-02T00:09:24Z",
  "ValidBefore": "2022-03-05T00:09:54Z",
  "Partition": "default",
  "Namespace": "default",
  "CreateIndex": 1527,
  "ModifyIndex": 1527
  }

< HTTP/2 200
< content-type: application/json
< date: Wed, 02 Mar 2022 03:27:45 GMT
< strict-transport-security: max-age=31536000
< vary: Accept-Encoding
< x-cache: MISS
< x-consul-default-acl-policy: deny
< x-consul-index: 1527

  {
  "SerialNumber": "2b:07:f4:f6:a5:3d:84:2e:fa:1a:27:8b:af:57:ba:1b:84:b1:1e:91",
  "CertPEM": "Edited for Brevity",
  "PrivateKeyPEM": "Edited for Brevity",
  "Service": "postgres",
  "ServiceURI": "spiffe://dc5c5a9f-706c-de1f-d10f-f3c0645c46bc.consul/ns/default/dc/dc1/svc/postgres",
  "ValidAfter": "2022-03-02T00:09:24Z",
  "ValidBefore": "2022-03-05T00:09:54Z",
  "Partition": "default",
  "Namespace": "default",
  "CreateIndex": 1527,
  "ModifyIndex": 1527
  }

The time difference between ValidAfter and ValidBefore should match the value of the configured TTL. In this case, the value of these keys is 72 hours and 30 seconds, matching the configured value of 72 hours, with 30 seconds set as a buffer by the CA.

»Cleanup:

Exit the workbench.

$ exit

$ exit

Run terraform destroy in both projects to tear down the environment

$ cd working-environment; terraform destroy -auto-approve && cd ../; export TF_VAR_run_cleanup=true; terraform destroy -auto-approve

$ cd working-environment; terraform destroy -auto-approve && cd ../; export TF_VAR_run_cleanup=true; terraform destroy -auto-approve

This can take approximately 15 minutes to complete.

»Next Steps

In this tutorial you configured Consul Connect service mesh's certification Authority using the PKI secrets engine in Vault. You configured a root certificate for Consul Connect backed by Vault, and generated leaf certificates for services deployed to Consul, signed by this root certificate.

Continue learning about the options of securing Consul by taking additional tutorials on Learn. The Secure Consul Gossip Communication with Vault tutorial teaches you to secure gossip protocol communication inside your Consul clusters. The Generate mTLS Certificates for Consul using Vault tutorial helps you implement mutual TLS for the overall Consul cluster.

Learn more about ACL token administration on Vault with the Generate Consul Tokens with HashiCorp Vault tutorial.

stdin: is not a tty