Application deployments in a Kubernetes cluster can leverage Vault to manage their secrets. Vault run internally is explored in the Vault Installation to Minikube via Helm and Injecting Secrets into Kubernetes Pods via Vault Helm Sidecar guides. There are situations where you may have an existing Vault service that is external to the cluster.
In this tutorial, you will run Vault locally, start a Kubernetes cluster with Minikube, deploy an application that retrieves secrets from this Vault, and configure an injector only deployment to inject secrets into the pods from this Vault.
»Prerequisites
This tutorial requires the Kubernetes command-line interface (CLI) and the Helm CLI installed, Minikube, Vault, and the Vault Helm chart, the sample web application, and additional configuration to bring it all together.
Online tutorial: An interactive tutorial is also available if you do not wish to install the following resources. Click the Show Terminal button to start.
This tutorial was last tested 11 Aug 2020 on a macOS 10.15.6 using this configuration.
Vault version.
$ vault version
Vault v1.4.1 ('b2b4ab9577e413b00d9b727e2c3f465561bd38bd')
Docker version.
$ docker version
Client: Docker Engine - Community
Version: 19.03.8
## ...
Minikube version.
$ minikube version
minikube version: v1.12.1
commit: 5664228288552de9f3a446ea4f51c6f29bbdd0e0
Helm version.
$ helm version
version.BuildInfo{Version:"v3.2.4", GitCommit:"0ad800ef43d3b826f31a5ad8dfbb4fe05d143688", GitTreeState:"dirty", GoVersion:"go1.14.3"}
These are recommended software versions and the output displayed may vary depending on your environment and the software versions you use.
First, follow the directions to install Minikube, including VirtualBox or similar.
Next, install kubectl CLI and helm CLI.
Install kubectl
with Homebrew.
$ brew install kubernetes-cli
Install helm
with Homebrew.
$ brew install helm
Next, retrieve the web application and additional configuration by cloning the hashicorp/vault-guides repository from GitHub.
$ git clone https://github.com/hashicorp/vault-guides.git
This repository contains supporting content for all of the Vault learn guides. The content specific to this tutorial can be found within a sub-directory.
Go into the
vault-guides/operations/provision-vault/kubernetes/minikube/external-vault
directory.
$ cd vault-guides/operations/provision-vault/kubernetes/minikube/external-vault
Working directory: This tutorial assumes that the remainder of commands are executed within this directory.
»Start Vault
Vault running external of a Kubernetes cluster can be addressed by any of its pods as long as the Vault server is network addressable. Running Vault locally alongside of Minikube is possible if the Vault server is bound to the same network as the cluster.
In another terminal, start a Vault dev server with root
as the root
token that listens for requests at 0.0.0.0:8200
.
$ vault server -dev -dev-root-token-id root -dev-listen-address 0.0.0.0:8200
Setting the -dev-listen-address
to 0.0.0.0:8200
overrides the default
address of a Vault dev server (127.0.0.1:8200
) and enables Vault to be
addressable by the Kubernetes cluster and its pods because it binds to a
shared network.
Insecure operation: Do not run a Vault dev server in production. This approach is only used here to simplify the unsealing process for this demonstration.
Export an environment variable for the vault
CLI to address the Vault server.
$ export VAULT_ADDR=http://0.0.0.0:8200
The web application that you deploy, expects Vault to store a username and
password stored at the path secret/devwebapp/config
. To create this secret
requires that a key-value secret
engine is enabled and a
username and password is put at the specified path. By default the Vault dev
server starts with a key-value secrets engine enabled at the path prefixed with
secret
.
Login with the root token.
$ vault login root
Success! You are now authenticated. The token information displayed below
is already stored in the token helper. You do NOT need to run "vault login"
again. Future Vault requests will automatically use this token.
Key Value
--- -----
token root
token_accessor 6NYAtL0ANmAGVmLX3tx4bVgu
token_duration ∞
token_renewable false
token_policies ["root"]
identity_policies []
policies ["root"]
Create a secret at path secret/devwebapp/config
with a username
and
password
.
$ vault kv put secret/devwebapp/config username='giraffe' password='salsa'
Key Value
--- -----
created_time 2020-08-11T16:59:42.076636Z
deletion_time n/a
destroyed false
version 1
Verify that the secret is defined at the path secret/data/devwebapp/config
.
$ vault read -format json secret/data/devwebapp/config | jq ".data.data"
{
"password": "salsa",
"username": "giraffe"
}
Learn more: This tutorial focuses on Vault's integration with Kubernetes and not interacting the key-value secrets engine. For more information refer to the Static Secrets: Key/Value Secret tutorial.
The Vault server, with secret, is ready to be addressed by a Kubernetes cluster and the pods deployed in it.
»Start Minikube
Minikube is a CLI tool that provisions and manages the lifecycle of single-node Kubernetes clusters locally inside Virtual Machines (VM) on your system.
Start a Kubernetes cluster.
$ minikube start --driver=docker
😄 minikube v1.12.1 on Darwin 10.15.6
✨ Using the docker driver based on user configuration
👍 Starting control plane node minikube in cluster minikube
🔥 Creating docker container (CPUs=2, Memory=1991MB) ...
🐳 Preparing Kubernetes v1.18.3 on Docker 19.03.2 ...
🔎 Verifying Kubernetes components...
🌟 Enabled addons: default-storageclass, storage-provisioner
🏄 Done! kubectl is now configured to use "minikube"
The initialization process takes several minutes as it retrieves any necessary dependencies and executes various container images.
Verify the status of the Minikube cluster.
$ minikube status
minikube
type: Control Plane
host: Running
kubelet: Running
apiserver: Running
kubeconfig: Configured
Additional waiting: Even if this last command completed successfully, you may have to wait for Minikube to be available. If an error is displayed, try again after a few minutes.
The host, kubelet, apiserver report that they are running. The kubectl
, a
command line interface (CLI) for running commands against Kubernetes cluster, is
also configured to communicate with this recently started cluster.
Minikube provides a visual representation of the status in a web-based dashboard. This interface displays the cluster activity in a visual interface that can be used to explore the issues affecting it.
In another terminal, launch the minikube dashboard.
$ minikube dashboard
The operating system's default browser opens and displays the dashboard.
»Determine the Vault address
A service bound to all networks on the host, as you configured Vault, is addressable by pods within Minikube's cluster by sending requests to the gateway address of the Kubernetes cluster.
Start a minikube SSH session.
$ minikube ssh
## ... minikube ssh login
Within this SSH session, retrieve the value of the Minikube host.
$ dig +short host.docker.internal
192.168.65.2
Docker networking: The host has a changing IP address (or none if you
have no network access). We recommend that you connect to the special DNS name
host.docker.internal which resolves to the internal IP address used by the host.
host.docker.internal
. This is for development purpose and will not work in
production. For more information, review the documentation for
Mac,
Windows.
Next, retrieve the status of the Vault server to verify network connectivity.
$ dig +short host.docker.internal | xargs -I{} curl -s http://{}:8200/v1/sys/seal-status
{
"type": "shamir",
"initialized": true,
"sealed": false,
"t": 1,
"n": 1,
"progress": 0,
"nonce": "",
"version": "1.5.0",
"migration": false,
"cluster_name": "vault-cluster-44ba824c",
"cluster_id": "adc0bb6a-e330-3e7a-e0c7-38061c3bf191",
"recovery_seal": false,
"storage_type": "inmem"
}
The output displays that Vault is initialized and unsealed. This confirms that pods within your cluster are able to reach Vault given that each pod is configured to use the gateway address.
Next, exit the Minikube SSH session.
$ exit
Finally, create a variable named EXTERNAL_VAULT_ADDR to capture the Minikube gateway address.
$ EXTERNAL_VAULT_ADDR=$(minikube ssh "dig +short host.docker.internal")
Verify that the variable contains the ip address you saw when executed in the minikube shell.
$ echo $EXTERNAL_VAULT_ADDR
192.168.65.2
»Deploy application with hard-coded Vault address
The most direct way for a pod within the cluster to address Vault is with a hard-coded network address defined within the application code or provided as an environment variable. We've created and published a web application that you will deploy with the Vault address overridden.
First, create a Kubernetes service account for the pods to use to authenticate.
$ cat <<EOF | kubectl apply -f -
apiVersion: v1
kind: ServiceAccount
metadata:
name: internal-app
EOF
Create a deployment with this web application that sets the VAULT_ADDR
to
EXTERNAL_VAULT_ADDR
.
$ cat <<EOF | kubectl apply -f -
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: devwebapp
labels:
app: devwebapp
spec:
replicas: 1
selector:
matchLabels:
app: devwebapp
template:
metadata:
labels:
app: devwebapp
spec:
serviceAccountName: internal-app
containers:
- name: app
image: burtlo/devwebapp-ruby:k8s
imagePullPolicy: Always
env:
- name: VAULT_ADDR
value: "http://$EXTERNAL_VAULT_ADDR:8200"
EOF
The web application, targeting the external Vault, is deployed as a pod within the default namespace.
Get all the pods within the default namespace.
$ kubectl get pods
NAME READY STATUS RESTARTS AGE
devwebapp-68cc55948b-w9745 1/1 Running 0 4m
Wait until the devwebapp
pod reports that is running and ready (1/1
).
Request content served at localhost:8080
from within the devwebapp
pod.
$ kubectl exec \
$(kubectl get pod -l app=devwebapp -o jsonpath="{.items[0].metadata.name}") \
-- curl -s localhost:8080 ; echo
The result displays the secret is defined at the path
secret/data/devwebapp/config
.
{"password"=>"salsa", "username"=>"giraffe"}
The web application authenticates with the external Vault server using the root
token and returns the secret defined at the path secret/data/devwebapp/config
.
This hard-coded approach is an effective solution if the address to the Vault
server does not change.
»Deploy service and endpoints to address an external Vault
An external Vault may not have a static network address that services within the cluster can rely upon. When Vault's network address changes each service also needs to change to continue its operation. Another approach to manage this network address is to define a Kubernetes service and endpoints.
A service creates an abstraction around pods or an external service. When an application running in a pod requests the service, that request is routed to the endpoints that share the service name.
Deploy a service named external-vault
and a corresponding endpoint configured
to address the EXTERNAL_VAULT_ADDR
.
$ cat <<EOF | kubectl apply -f -
---
apiVersion: v1
kind: Service
metadata:
name: external-vault
namespace: default
spec:
ports:
- protocol: TCP
port: 8200
---
apiVersion: v1
kind: Endpoints
metadata:
name: external-vault
subsets:
- addresses:
- ip: $EXTERNAL_VAULT_ADDR
ports:
- port: 8200
EOF
Verify that the external-vault
service is addressable from within the
devwebapp
pod.
$ kubectl exec \
$(kubectl get pod -l app=devwebapp -o jsonpath="{.items[0].metadata.name}") \
-- curl -s http://external-vault:8200/v1/sys/seal-status | jq
The result displays the status of the Vault server.
{
"type": "shamir",
"initialized": true,
"sealed": false,
"t": 1,
"n": 1,
"progress": 0,
"nonce": "",
"version": "1.5.0",
"migration": false,
"cluster_name": "vault-cluster-44ba824c",
"cluster_id": "adc0bb6a-e330-3e7a-e0c7-38061c3bf191",
"recovery_seal": false,
"storage_type": "inmem"
}
Next, create a deployment that sets the VAULT_ADDR
to the external-vault
service.
$ kubectl apply -f deployment-01-external-vault-service.yml
deployment.apps/devwebapp-through-service created
This deployment named devwebapp-through-service
creates a pod that addresses
Vault through the service instead of the hard-coded network address.
Get all the pods within the default namespace.
$ kubectl get pods
NAME READY STATUS RESTARTS AGE
devwebapp-54b89c546b-zd8np 1/1 Running 0 36m
devwebapp-through-service-6b4b79994-9t7v7 1/1 Running 0 20s
Wait until the devwebapp-through-service
pod is running and ready (1/1
).
Finally, request content served at localhost:8080
from within the
devwebapp-through-service
pod.
$ kubectl exec \
$(kubectl get pod -l app=devwebapp-through-service -o jsonpath="{.items[0].metadata.name}") \
-- curl -s localhost:8080 ; echo
The result displays the secret is defined at the path
secret/data/devwebapp/config
.
{"password"=>"salsa", "username"=>"giraffe"}
The web application authenticates and requests the secret from the external
Vault server that it found through the external-vault
service.
»Install the Vault Helm chart configured to address an external Vault
The Vault Helm chart can deploy only the Vault Agent Injector service configured to target an external Vault. The injector service enables the authentication and secret retrieval for the applications, by adding Vault Agent containers as they are written to the pod automatically when a deployment includes specific annotations.
In this section, you will create a Kubernetes service account; configure Vault's Kubernetes authentication and create a role to access a secret; install the Vault Helm chart to run only the injector service; and patch a deployment.
»Define a Kubernetes service account
Create a service account, secret, and ClusterRoleBinding with the necessary permissions to allow Vault to perform token reviews with Kubernetes.
$ cat <<EOF | kubectl create -f -
---
apiVersion: v1
kind: ServiceAccount
metadata:
name: vault-auth
---
apiVersion: v1
kind: Secret
metadata:
name: vault-auth
annotations:
kubernetes.io/service-account.name: vault-auth
type: kubernetes.io/service-account-token
---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRoleBinding
metadata:
name: role-tokenreview-binding
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: system:auth-delegator
subjects:
- kind: ServiceAccount
name: vault-auth
namespace: default
EOF
This creates the vault-auth
service account, the vault-auth
secret, and
the ClusterRoleBinding that uses the created service account.
»Configure Kubernetes authentication
Vault provides a Kubernetes authentication method that enables clients to authenticate with a Kubernetes Service Account Token.
Enable the Kubernetes authentication method.
$ vault auth enable kubernetes
Success! Enabled kubernetes auth method at: kubernetes/
Vault accepts this service token from any client within the Kubernetes cluster. During authentication, Vault verifies that the service account token is valid by querying a configured Kubernetes endpoint. To configure it correctly requires capturing the JSON web token (JWT) for the service account, the Kubernetes CA certificate, and the Kubernetes host URL.
First, get the JSON web token (JWT) for this service account.
$ TOKEN_REVIEW_JWT=$(kubectl get secret vault-auth -o go-template='{{ .data.token }}' | base64 --decode)
Next, retrieve the Kubernetes CA certificate.
$ KUBE_CA_CERT=$(kubectl config view --raw --minify --flatten -o jsonpath='{.clusters[].cluster.certificate-authority-data}' | base64 --decode)
Next, retrieve the Kubernetes host URL.
$ KUBE_HOST=$(kubectl config view --raw --minify --flatten -o jsonpath='{.clusters[].cluster.server}')
Finally, configure the Kubernetes authentication method to use the service account token, the location of the Kubernetes host, and its certificate.
$ vault write auth/kubernetes/config \
token_reviewer_jwt="$TOKEN_REVIEW_JWT" \
kubernetes_host="$KUBE_HOST" \
kubernetes_ca_cert="$KUBE_CA_CERT"
For a Vault client to read the secret data defined in the Start
Vault section requires that the read capability be granted for
the path secret/data/devwebapp/config
.
Write out the policy named devwebapp
that enables the read
capability
for secrets at path secret/data/devwebapp/config
$ vault policy write devwebapp - <<EOF
path "secret/data/devwebapp/config" {
capabilities = ["read"]
}
EOF
Create a Kubernetes authentication role named devweb-app
.
$ vault write auth/kubernetes/role/devweb-app \
bound_service_account_names=internal-app \
bound_service_account_namespaces=default \
policies=devwebapp \
ttl=24h
The role connects the Kubernetes service account, internal-app
, and namespace,
default
, with the Vault policy, devwebapp
. The tokens returned after
authentication are valid for 24 hours.
»Install the Vault Helm chart
The Vault Helm chart is able to install only the Vault Agent Injector service.
Add the HashiCorp Helm repository.
$ helm repo add hashicorp https://helm.releases.hashicorp.com
"hashicorp" has been added to your repositories
Install the latest version of the Vault server running in external mode.
$ helm install vault hashicorp/vault \
--set "injector.externalVaultAddr=http://external-vault:8200"
The Vault Agent Injector pod is deployed in the default namespace.
Get all the pods in the default namespace.
$ kubectl get pods
NAME READY STATUS RESTARTS AGE
devwebapp-54b89c546b-zd8np 1/1 Running 0 84m
devwebapp-through-service-6b4b79994-9t7v7 1/1 Running 0 48m
vault-agent-injector-7b6cd469d8-8svg5 1/1 Running 0 15s
Wait until the vault-agent-injector
pod reports that it is running and ready
(1/1
).
»Inject secrets into the pod
The Vault Agent Injector only modifies a deployment if it contains a specific set of annotations. An existing deployment may have its definition patched to include the necessary annotations.
Display the deployment patch patch-02-inject-secrets.yml
.
$ cat patch-02-inject-secrets.yml
spec:
template:
metadata:
annotations:
vault.hashicorp.com/agent-inject: "true"
vault.hashicorp.com/role: "devweb-app"
vault.hashicorp.com/agent-inject-secret-credentials.txt: "secret/data/devwebapp/config"
These
annotations
define a partial structure of the deployment schema and are prefixed with
vault.hashicorp.com
.
agent-inject
enables the Vault Agent Injector servicerole
is the Vault Kubernetes authentication roleagent-inject-secret-FILEPATH
prefixes the path of the file,credentials.txt
written to the/vault/secrets
directory. The value is the path to the secret defined in Vault.
Patch the existing devwebapp
deployment with the annotations to write the
secrets to the pod.
$ kubectl patch deployment devwebapp --patch "$(cat patch-02-inject-secrets.yml)"
deployment.apps/devwebapp patched
A new devwebapp
pod starts alongside the existing pod. When it is ready the
original terminates and removes itself from the list of active pods.
Get all the pods within the default namespace.
$ kubectl get pods
NAME READY STATUS RESTARTS AGE
devwebapp-5ddc869888-xbwfj 0/2 Init:0/1 0 14m
devwebapp-through-service-6b4b79994-9t7v7 1/1 Running 0 48m
vault-agent-injector-7b6cd469d8-8svg5 1/1 Running 0 17m
Wait until the re-deployed devwebapp
pod reports that it is running and ready
(2/2
).
The Vault Agent Injector service automatically writes the secrets to the
devwebapp
pod at the filepath /vault/secrets/credentials.txt
.
Display the secrets written to the file /vault/secrets/secret-credentials.txt
on the devwebapp
pod.
$ kubectl exec -it \
$(kubectl get pod -l app=devwebapp -o jsonpath="{.items[0].metadata.name}") \
-c app -- cat /vault/secrets/credentials.txt
The result displays the unformatted secret data present on the container.
data: map[password:salsa username:giraffe]
metadata: map[created_time:2019-12-20T18:17:50.930264759Z deletion_time: destroyed:false version:2]
The unformatted secret data is present on the container.
Formatting data: A template can be applied to structure this data to meet the needs of the application.
»Next Steps
You deployed Vault external to a Kubernetes cluster and deployed pods that leveraged it as a secrets store. First, through a hard-coded network address. Second, aliased behind a Kubernetes service and endpoint. And finally, through the Vault Helm's chart and the injector service with annotations applied to a deployment. Learn more about the Vault Helm chart by reading the documentation, exploring the project source code, exploring how pods can retrieve secrets through the Vault Injector service via annotations, or secrets mounted on ephemeral volumes.