Kubernetes configured to use Vault as a certificate manager enables your services to establish their identity and communicate securely over the network with other services or clients internal or external to the cluster.
Jetstack's cert-manager enables Vault's PKI secrets engine to dynamically generate X.509 certificates within Kubernetes through an Issuer interface.
In this tutorial, you setup Vault with the Vault Helm chart, configure the PKI secrets engine and Kubernetes authentication. Then install Jetstack's cert-manager, configure it to use Vault, and request a certificate.
»Prerequisites
This tutorial requires the Kubernetes command-line interface (CLI) and the Helm CLI installed, Minikube, the Vault Helm charts, the sample web application, and additional configuration to bring it all together.
Online tutorial: An interactive tutorial is also available if you do not wish to install the following resources. Click the Show Terminal button to start.
This tutorial was last tested 29 Apr 2020 on a macOS 10.15.4 using this configuration.
Docker version.
$ docker version
Client: Docker Engine - Community
Version: 19.03.8
## ...
Minikube version.
$ minikube version
minikube version: v1.8.2
commit: eb13446e786c9ef70cb0a9f85a633194e62396a1
Helm version.
$ helm version
version.BuildInfo{Version:"v3.1.2", GitCommit:"d878d4d45863e42fd5cff6743294a11d28a9abce", GitTreeState:"clean", GoVersion:"go1.14"}
These are recommended software versions and the output displayed may vary depending on your environment and the software versions you use.
First, follow the directions for installing Minikube, including VirtualBox or similar.
Next, install kubectl CLI and helm CLI.
Install kubectl
with Homebrew.
$ brew install kubernetes-cli
Install helm
with Homebrew.
$ brew install helm
Next, retrieve the web application and additional configuration by cloning the hashicorp/vault-guides repository from GitHub.
$ git clone https://github.com/hashicorp/vault-guides.git
This repository contains supporting content for all of the Vault learn guides. The content specific to this tutorial can be found within a sub-directory.
Go into the
vault-guides/operations/provision-vault/kubernetes/minikube/external-vault
directory.
$ cd vault-guides/operations/provision-vault/kubernetes/minikube/external-vault
Working directory: This tutorial assumes that the remainder of commands are executed within this directory.
»Start Minikube
Minikube is a CLI tool that provisions and manages the lifecycle of single-node Kubernetes clusters. These clusters are run locally inside Virtual Machines (VM).
Start a Kubernetes cluster.
$ minikube start
😄 minikube v1.8.2 on Darwin 10.15.4
✨ Automatically selected the hyperkit driver
🔥 Creating hyperkit VM (CPUs=2, Memory=4000MB, Disk=20000MB) ...
🐳 Preparing Kubernetes v1.17.3 on Docker 19.03.6 ...
🚀 Launching Kubernetes ...
🌟 Enabling addons: default-storageclass, storage-provisioner
⌛ Waiting for cluster to come online ...
🏄 Done! kubectl is now configured to use "minikube"
Verify the status of the Minikube cluster.
$ minikube status
host: Running
kubelet: Running
apiserver: Running
kubeconfig: Configured
Additional waiting: Even if the previous step completed successfully, you may have to wait for Minikube to be available. If you see an error, try again after a few minutes.
The host, kubelet, and apiserver report that they are running. The kubectl
, a
command line interface (CLI) for running commands against Kubernetes cluster, is
also configured to communicate with this recently started cluster.
»Install the Vault Helm chart
Add the HashiCorp Helm repository.
$ helm repo add hashicorp https://helm.releases.hashicorp.com
"hashicorp" has been added to your repositories
Install the latest version of the Vault server running in standalone mode with the Vault Agent Injector service disabled.
$ helm install vault hashicorp/vault --set "injector.enabled=false"
NAME: vault
## ...
The Vault server runs in standalone mode on a single pod. By default the Helm
chart starts a Vault Agent Injector pod but that is disabled
injector.enabled=false
.
Get all the pods within the default namespace.
$ kubectl get pods
NAME READY STATUS RESTARTS AGE
vault-0 0/1 Running 0 87s
The vault-0
pod is deployed. The Vault server running in the pod's container
reports that it is running but it is not ready (0/1
). To ready the pod
requires that the Vault server is initialized and unsealed.
Get all the services within the default namespace.
$ kubectl get service
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
kubernetes ClusterIP 10.96.0.1 <none> 443/TCP 9m5s
vault ClusterIP 10.105.21.154 <none> 8200/TCP,8201/TCP 2m34s
The Vault Helm chart creates a Service that directs requests to the Vault pod.
This enables us to address the Vault server within the cluster with the address
http://vault.default:8200
.
»Initialize and unseal Vault
Vault run in standalone mode starts uninitialized and in the sealed state. Prior to initialization the storage backend is not prepared to receive data.
Initialize Vault with one key share and one key threshold.
$ kubectl exec vault-0 -- vault operator init -key-shares=1 -key-threshold=1 \
-format=json > init-keys.json
The operator init
command
generates a master key that it disassembles into key shares -key-shares=1
and
then sets the number of key shares required to unseal Vault -key-threshold=1
.
These key shares are written to the output as unseal keys in JSON format
-format=json
. Here the output is redirected to a local file named
init-keys.json
View the unseal key found in init-keys.json
.
$ cat init-keys.json | jq -r ".unseal_keys_b64[]"
hmeMLoRiX/trBTx/xPZHjCcZ7c4H8OCt2Njkrv2yXZY=
Insecure operation: Do not run an unsealed Vault in production with a single key share and a single key threshold. This approach is only used here to simplify the unsealing process for this demonstration.
Create a variable named VAULT_UNSEAL_KEY
to capture the Vault unseal key.
$ VAULT_UNSEAL_KEY=$(cat init-keys.json | jq -r ".unseal_keys_b64[]")
After initialization, Vault is configured to know where and how to access the storage, but does not know how to decrypt any of it. Unsealing is the process of constructing the master key necessary to read the decryption key to decrypt the data, allowing access to the Vault.
Unseal Vault running on the vault-0
pod with the $VAULT_UNSEAL_KEY
.
$ kubectl exec vault-0 -- vault operator unseal $VAULT_UNSEAL_KEY
Key Value
--- -----
Seal Type shamir
Initialized true
Sealed false
Total Shares 1
Threshold 1
Version 1.4.0
Cluster Name vault-cluster-3ecefcf2
Cluster ID 0998f747-af15-f994-cded-295836b718d6
HA Enabled false
The operator unseal
command reports that Vault is initialized and unsealed.
Insecure operation: Providing the unseal key with the command writes the key to your shell's history. This approach is only used here to simplify the unsealing process for this demonstration.
Get all the pods within the default namespace.
$ kubectl get pods
NAME READY STATUS RESTARTS AGE
vault-0 1/1 Running 0 5m49s
The vault-0
pod reports that it is ready 1/1
. Vault is ready for you to
login with the root token generated during the initialization.
View the root token found in init-keys.json
.
$ cat init-keys.json | jq -r ".root_token"
s.XzExf8TjRVYKm85xMATa6Q7U
Create a variable named VAULT_ROOT_TOKEN
to capture the root token.
$ VAULT_ROOT_TOKEN=$(cat init-keys.json | jq -r ".root_token")
Login to Vault running on the vault-0
pod with the $VAULT_ROOT_TOKEN
.
$ kubectl exec vault-0 -- vault login $VAULT_ROOT_TOKEN
Success! You are now authenticated. The token information displayed below
is already stored in the token helper. You do NOT need to run "vault login"
again. Future Vault requests will automatically use this token.
Key Value
--- -----
token s.P3Koh6BZikQPDxPSNwDzmKJ5
token_accessor kHFYypyS2EcYpMyrsyXUQmNa
token_duration ∞
token_renewable false
token_policies ["root"]
identity_policies []
policies ["root"]
The Vault server is ready to be configured as a certificate store.
»Configure PKI secrets engine
First, start an interactive shell session on the vault-0
pod.
$ kubectl exec -it vault-0 -- /bin/sh
/ $
Your system prompt is replaced with a new prompt / $
. Commands issued at this
prompt are executed on the vault-0
container.
Enable the PKI secrets engine at its default path.
$ vault secrets enable pki
Success! Enabled the pki secrets engine at: pki/
By default the KPI secrets engine sets the time-to-live (TTL) to 30 days. A certificate can have its lease extended to ensure certificate rotation on a yearly basis (8760h).
Configure the max lease time-to-live (TTL) to 8760h
.
$ vault secrets tune -max-lease-ttl=8760h pki
Success! Tuned the secrets engine at: pki/
Vault can accept an existing key pair, or it can generate its own self-signed root. In general, we recommend maintaining your root CA outside of Vault and providing Vault a signed intermediate CA.
Generate a self-signed certificate valid for 8760h
.
$ vault write pki/root/generate/internal \
common_name=example.com \
ttl=8760h
Key Value
--- -----
certificate -----BEGIN CERTIFICATE-----
## ...
-----END CERTIFICATE-----
expiration 1619120269
issuing_ca -----BEGIN CERTIFICATE-----
## ...
-----END CERTIFICATE-----
serial_number 65:37:b5:b3:91:6c:7b:d8:33:22:03:28:b1:58:ff:be:8a:72:a4:c0
Configure the PKI secrets engine certificate issuing and certificate revocation list (CRL) endpoints to use the Vault service in the default namespace.
$ vault write pki/config/urls \
issuing_certificates="http://vault.default:8200/v1/pki/ca" \
crl_distribution_points="http://vault.default:8200/v1/pki/crl"
Success! Data written to: pki/config/urls
Configure a role named example-dot-com
that enables the creation of
certificates example.com
domain with any subdomains.
$ vault write pki/roles/example-dot-com \
allowed_domains=example.com \
allow_subdomains=true \
max_ttl=72h
Success! Data written to: pki/roles/example-dot-com
The role, example-dot-com
, is a logical name that maps to a policy used to
generate credentials. This generates a number of endpoints that are used by the
Kubernetes service account to issue and sign these certificates. A policy must
be created that enables these paths.
Create a policy named pki
that enables read access to the PKI secrets engine
paths.
$ vault policy write pki - <<EOF
path "pki*" { capabilities = ["read", "list"] }
path "pki/roles/example-dot-com" { capabilities = ["create", "update"] }
path "pki/sign/example-dot-com" { capabilities = ["create", "update"] }
path "pki/issue/example-dot-com" { capabilities = ["create"] }
EOF
Success! Uploaded policy: pki
These paths enable the token to view all the roles created for this PKI secrets
engine and access the sign
and issues
operations for the example-dot-com
role.
Lastly, exit the vault-0
pod.
$ exit
»Configure Kubernetes authentication
Vault provides a Kubernetes authentication method that enables clients to authenticate with a Kubernetes Service Account Token.
First, start an interactive shell session on the vault-0
pod.
$ kubectl exec -it vault-0 -- /bin/sh
/ $
Your system prompt is replaced with a new prompt / $
. Commands issued at this
prompt are executed on the vault-0
container.
Enable the Kubernetes authentication method.
$ vault auth enable kubernetes
Success! Enabled kubernetes auth method at: kubernetes/
Configure the Kubernetes authentication method to use the service account token, the location of the Kubernetes host, and its certificate.
$ vault write auth/kubernetes/config \
token_reviewer_jwt="$(cat /var/run/secrets/kubernetes.io/serviceaccount/token)" \
kubernetes_host="https://$KUBERNETES_PORT_443_TCP_ADDR:443" \
kubernetes_ca_cert=@/var/run/secrets/kubernetes.io/serviceaccount/ca.crt
Success! Data written to: auth/kubernetes/config
The token_reviewer_jwt
and kubernetes_ca_cert
reference files written to the
container by Kubernetes. The environment variable KUBERNETES_PORT_443_TCP_ADDR
references the internal network address of the Kubernetes host.
Finally, create a Kubernetes authentication role named issuer
that binds
the pki
policy with a Kubernetes service account named issuer
.
$ vault write auth/kubernetes/role/issuer \
bound_service_account_names=issuer \
bound_service_account_namespaces=default \
policies=pki \
ttl=20m
Success! Data written to: auth/kubernetes/role/issuer
The role connects the Kubernetes service account, issuer
, in the default
namespace with the pki
Vault policy. The tokens returned after authentication
are valid for 20 minutes. This Kubernetes service account name, issuer
, is
created in the Deploy Issuer and Certificate
section.
Lastly, exit the vault-0
pod.
$ exit
»Deploy Cert Manager
Jetstack's cert-manager is a Kubernetes add-on that automates the management and issuance of TLS certificates from various issuing sources. Vault can be configured as one of those sources. The cert-manager requires the creation of a set of Kubernetes resources that provide the interface to the certificate creation.
Install Jetstack's cert-manager's version 0.14.3 resources.
$ kubectl apply --validate=false -f https://github.com/jetstack/cert-manager/releases/download/v0.14.3/cert-manager.crds.yaml
customresourcedefinition.apiextensions.k8s.io/challenges.acme.cert-manager.io created
customresourcedefinition.apiextensions.k8s.io/orders.acme.cert-manager.io created
customresourcedefinition.apiextensions.k8s.io/certificaterequests.cert-manager.io created
customresourcedefinition.apiextensions.k8s.io/certificates.cert-manager.io created
customresourcedefinition.apiextensions.k8s.io/clusterissuers.cert-manager.io created
customresourcedefinition.apiextensions.k8s.io/issuers.cert-manager.io created
Create a namespace named cert-manager
to host the cert-manager.
$ kubectl create namespace cert-manager
namespace/cert-manager created
Jetstack's cert-manager Helm chart is available in a repository that they maintain. Helm can request and install Helm charts from these custom repositories.
Add the jetstack
chart repository.
$ helm repo add jetstack https://charts.jetstack.io
"jetstack" has been added to your repositories
Helm maintains a cached list of charts for every repository that it maintains. This list needs to be updated periodically so that Helm knows about all available charts and their releases. A repository recently added needs to be updated before any chart is requested.
Update the local list of Helm charts.
$ helm repo update
Hang tight while we grab the latest from your chart repositories...
...Successfully got an update from the "jetstack" chart repository
Update Complete. ⎈ Happy Helming!⎈
The results show that the jetstack
chart repository has retrieved an update.
Install the cert-manager chart version 0.11 in the cert-manager
namespace.
$ helm install cert-manager \
--namespace cert-manager \
--version v0.14.3 \
jetstack/cert-manager
NAME: cert-manager
## ...
The cert-manager chart deploys a number of pods within the cert-manager
namespace.
Get all the pods within the cert-manager
namespace.
$ kubectl get pods --namespace cert-manager
NAME READY STATUS RESTARTS AGE
cert-manager-66958f45fc-pdf64 1/1 Running 0 27s
cert-manager-cainjector-755bbf9c6b-gpgtg 1/1 Running 0 27s
cert-manager-webhook-76954fcbcd-w4lll 1/1 Running 0 27s
Wait until the pods prefixed with cert-manager
are running and ready (1/1
).
Thes pods now require configuration to interface with Vault.
»Configure an issuer and generate a certificate
The cert-manager enables you to define Issuers that interface with the Vault certificate generating endpoints. These Issuers are invoked when a Certificate is created.
When you configured Vault's Kubernetes
authentication a Kubernetes service
account, named issuer
, was granted the policy, named pki
, to the certificate
generation endpoints.
Create a service account named issuer
within the default namespace.
$ kubectl create serviceaccount issuer
serviceaccount/issuer created
The service account generated a secret that is required by the Issuer.
Get all the secrets in the default namespace.
$ kubectl get secrets
default-token-mlm2n kubernetes.io/service-account-token 3 13d
issuer-token-lmzpj kubernetes.io/service-account-token 3 47s
sh.helm.release.v1.vault.v1 helm.sh/release.v1 1 28m
vault-token-749nd kubernetes.io/service-account-token 3 28m
The issuer secret is displayed here as the secret prefixed with issuer-token
.
Create a variable named ISSUER_SECRET_REF
to capture the secret name.
$ ISSUER_SECRET_REF=$(kubectl get serviceaccount issuer -o json | jq -r ".secrets[].name")
Create an Issuer, named vault-issuer
, that defines Vault as a certificate
issuer.
$ cat <<EOF | kubectl apply -f -
apiVersion: cert-manager.io/v1alpha2
kind: Issuer
metadata:
name: vault-issuer
namespace: default
spec:
vault:
server: http://vault.default
path: pki/sign/example-dot-com
auth:
kubernetes:
mountPath: /v1/auth/kubernetes
role: issuer
secretRef:
name: $ISSUER_SECRET_REF
key: token
EOF
issuer.cert-manager.io/vault-issuer created
The specification defines the signing endpoint and the authentication endpoint and credentials.
metadata.name
sets the name of the Issuer tovault-issuer
spec.vault.server
sets the server address to the Kubernetes service created in the default namespacespec.vault.path
is the signing endpoint created by Vault's PKIexample-dot-com
rolespec.vault.auth.kubernetes.mountPath
sets the Vault authentication endpointspec.vault.auth.kubernetes.role
sets the Vault Kubernetes role toissuer
spec.vault.auth.kubernetes/secretRef.name
sets the secret for the Kubernetes service accountspec.vault.auth.kubernetes/secretRef.key
sets the type totoken
.
Generate a certificate named example-com
.
$ cat <<EOF | kubectl apply -f -
apiVersion: cert-manager.io/v1alpha2
kind: Certificate
metadata:
name: example-com
namespace: default
spec:
secretName: example-com-tls
issuerRef:
name: vault-issuer
commonName: www.example.com
dnsNames:
- www.example.com
EOF
certificate.cert-manager.io/example-com created
The Certificate, named example-com
, requests from Vault the certificate
through the Issuer, named vault-issuer
. The common name and DNS names are
names within the allowed domains for the configured Vault endpoint.
View the details of the example-com
certificate.
$ kubectl describe certificate.cert-manager example-com
Name: example-com
Namespace: default
## ...
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal GeneratedKey 10m cert-manager Generated a new private key
Normal Requested 10m cert-manager Created new CertificateRequest resource "example-com-1072521490"
Normal Issued 70s cert-manager Certificate issued successfully
The certificate reports that it has been issued successfully.
»Next steps
In this tutorial, you installed Vault configured the PKI secrets engine and Kubernetes authentication. Then installed Jetstack's cert-manager, configured it to use Vault, and requested a certificate.
Besides creation, these certificates can be revoked and removed. Learn more about Jetstack's cert-manager used in this tutorial and explore Vault's KPI secrets engine as a certificate authority in the Build Your Own Certificate Authority.