Integrate a Kubernetes Cluster with an External Vault

  • 15 min
  • Products Usedvault
  • This tutorial also appears in: Interactive.

Vault can manage secrets for Kubernetes application pods from outside the cluster. This could be HashiCorp Cloud Platform (HCP) Vault or another Vault service within your organization.

In this tutorial, you will run Vault locally, start a Kubernetes cluster with Minikube, deploy an application that retrieves secrets directly from Vault, through a Kubernetes service, and through secret injection via Vault Agent Injector.

»Prerequisites

This tutorial requires the Kubernetes command-line interface (CLI) and the Helm CLI installed, Minikube, Vault, and the Vault Helm chart, the sample web application, and additional configuration to bring it all together.

Online tutorial: An interactive tutorial is also available if you do not wish to install the following resources. Click the Show Terminal button to start.

This tutorial was last tested 03 Jun 2021 on a macOS 11.4 using this configuration.

Vault version.

$ vault version
Vault v1.7.2 (db0e4245d5119b5929e611ea4d9bf66e47f3f208)

Docker version.

$ docker version

Client: Docker Engine - Community
 Version:           20.10.6
 ## ...

Minikube version.

$ minikube version
minikube version: v1.20.0
commit: c61663e942ec43b20e8e70839dcca52e44cd85ae

Helm version.

$ helm version
version.BuildInfo{Version:"v3.5.4", GitCommit:"1b5edb69df3d3a08df77c9902dc17af864ff05d1", GitTreeState:"dirty", GoVersion:"go1.16.3"}

These are recommended software versions and the output displayed may vary depending on your environment and the software versions you use.

First, follow the directions to install Minikube, including VirtualBox or similar.

Next, install kubectl CLI and helm CLI.

Install kubectl with Homebrew.

$ brew install kubernetes-cli

Install helm with Homebrew.

$ brew install helm

Next, retrieve the web application and additional configuration by cloning the hashicorp/vault-guides repository from GitHub.

$ git clone https://github.com/hashicorp/vault-guides.git

This repository contains supporting content for all of the Vault learn guides. The content specific to this tutorial can be found in a sub-directory.

Go into the vault-guides/operations/provision-vault/kubernetes/minikube/external-vault directory.

$ cd vault-guides/operations/provision-vault/kubernetes/minikube/external-vault

»Start Vault

Vault running external of a Kubernetes cluster can be addressed by any of its pods as long as the Vault server is network addressable. Running Vault locally alongside of Minikube is possible if the Vault server is bound to the same network as the cluster.

In another terminal, start a Vault dev server with root as the root token that listens for requests at 0.0.0.0:8200.

$ vault server -dev -dev-root-token-id root -dev-listen-address 0.0.0.0:8200

Setting the -dev-listen-address to 0.0.0.0:8200 overrides the default address of a Vault dev server (127.0.0.1:8200) and enables Vault to be addressable by the Kubernetes cluster and its pods because it binds to a shared network.

Export an environment variable for the vault CLI to address the Vault server.

$ export VAULT_ADDR=http://0.0.0.0:8200

The web application that you deploy, expects Vault to store a username and password stored at the path secret/devwebapp/config. To create this secret requires that a key-value secret engine is enabled and a username and password is put at the specified path. By default the Vault dev server starts with a key-value secrets engine enabled at the path prefixed with secret.

Login with the root token.

$ vault login root
Success! You are now authenticated. The token information displayed below
is already stored in the token helper. You do NOT need to run "vault login"
again. Future Vault requests will automatically use this token.

Key                  Value
---                  -----
token                root
token_accessor       6NYAtL0ANmAGVmLX3tx4bVgu
token_duration       ∞
token_renewable      false
token_policies       ["root"]
identity_policies    []
policies             ["root"]

Create a secret at path secret/devwebapp/config with a username and password.

$ vault kv put secret/devwebapp/config username='giraffe' password='salsa'
Key              Value
---              -----
created_time     2020-08-11T16:59:42.076636Z
deletion_time    n/a
destroyed        false
version          1

Verify that the secret is stored at the path secret/devwebapp/config.

$ vault kv get -format=json secret/devwebapp/config | jq ".data.data"
{
  "password": "salsa",
  "username": "giraffe"
}

The Vault server, with secret, is ready to be addressed by a Kubernetes cluster and the pods deployed in it.

»Start Minikube

Minikube is a CLI tool that provisions and manages the lifecycle of single-node Kubernetes clusters locally inside Virtual Machines (VM) on your system.

Start a Kubernetes cluster.

$ minikube start --driver=docker
minikube v1.20.0 on Darwin 11.4 (arm64)
✨  Using the docker driver based on user configuration
👍  Starting control plane node minikube in cluster minikube
🚜  Pulling base image ...
🔥  Creating docker container (CPUs=2, Memory=1988MB) ...
🐳  Preparing Kubernetes v1.20.2 on Docker 20.10.6 ...
    ▪ Generating certificates and keys ...
    ▪ Booting up control plane ...
    ▪ Configuring RBAC rules ...
🔎  Verifying Kubernetes components...
    ▪ Using image gcr.io/k8s-minikube/storage-provisioner:v5
🌟  Enabled addons: storage-provisioner, default-storageclass
🏄  Done! kubectl is now configured to use "minikube" cluster and "default" namespace by default

The initialization process takes several minutes as it retrieves any necessary dependencies and executes various container images.

Verify the status of the Minikube cluster.

$ minikube status
minikube
type: Control Plane
host: Running
kubelet: Running
apiserver: Running
kubeconfig: Configured

The host, kubelet, apiserver report that they are running. The kubectl, a command line interface (CLI) for running commands against Kubernetes cluster, is also configured to communicate with this recently started cluster.

Minikube provides a visual representation of the status in a web-based dashboard. This interface displays the cluster activity in a visual interface that can be used to explore the issues affecting it.

In another terminal, launch the minikube dashboard.

$ minikube dashboard

The operating system's default browser opens and displays the dashboard.

»Determine the Vault address

A service bound to all networks on the host, as you configured Vault, is addressable by pods in Minikube's cluster by sending requests to the gateway address of the Kubernetes cluster.

Start a minikube SSH session.

$ minikube ssh
## ... minikube ssh login

Within this SSH session, retrieve the value of the Minikube host.

$ dig +short host.docker.internal
192.168.65.2

Next, retrieve the status of the Vault server to verify network connectivity.

$ dig +short host.docker.internal | xargs -I{} curl -s http://{}:8200/v1/sys/seal-status
{
  "type": "shamir",
  "initialized": true,
  "sealed": false,
  "t": 1,
  "n": 1,
  "progress": 0,
  "nonce": "",
  "version": "1.5.0",
  "migration": false,
  "cluster_name": "vault-cluster-44ba824c",
  "cluster_id": "adc0bb6a-e330-3e7a-e0c7-38061c3bf191",
  "recovery_seal": false,
  "storage_type": "inmem"
}

The output displays that Vault is initialized and unsealed. This confirms that pods in the cluster are able to reach Vault given that each pod is configured to use the gateway address.

Next, exit the Minikube SSH session.

$ exit

Finally, create a variable named EXTERNAL_VAULT_ADDR to capture the Minikube gateway address.

$ EXTERNAL_VAULT_ADDR=$(minikube ssh "dig +short host.docker.internal" | tr -d '\r')

Verify that the variable contains the ip address you saw when executed in the minikube shell.

$ echo $EXTERNAL_VAULT_ADDR
192.168.65.2

»Deploy application with hard-coded Vault address

The most direct way for a pod in the cluster to address Vault is with a hard-coded network address defined in the application code or provided as an environment variable. We've created and published a web application that allows override of the Vault address.

Create a Kubernetes service account named internal-app.

$ kubectl create sa internal-app

Define a pod named devwebapp with the web application that sets the VAULT_ADDR to EXTERNAL_VAULT_ADDR.

$ cat > devwebapp.yaml <<EOF
apiVersion: v1
kind: Pod
metadata:
  name: devwebapp
  labels:
    app: devwebapp
spec:
  serviceAccountName: internal-app
  containers:
    - name: app
      image: burtlo/devwebapp-ruby:k8s
      env:
      - name: VAULT_ADDR
        value: "http://$EXTERNAL_VAULT_ADDR:8200"
      - name: VAULT_TOKEN
        value: root
EOF

Create the devwebapp pod.

$ kubectl apply --filename devwebapp.yaml

Get all the pods in the default namespace.

$ kubectl get pods
NAME        READY   STATUS    RESTARTS   AGE
devwebapp   1/1     Running   0          4m

Wait until the devwebapp pod reports that is running and ready (1/1).

Request content served at localhost:8080 from within the devwebapp pod.

$ kubectl exec devwebapp -- curl -s localhost:8080 ; echo

The result displays the secret is defined at the path secret/data/devwebapp/config.

{"password"=>"salsa", "username"=>"giraffe"}

The web application authenticates with the external Vault server using the root token and returns the secret defined at the path secret/data/devwebapp/config. This hard-coded approach is an effective solution if the address to the Vault server does not change.

»Deploy service and endpoints to address an external Vault

An external Vault may not have a static network address that services in the cluster can rely upon. When Vault's network address changes each service also needs to change to continue its operation. Another approach to manage this network address is to define a Kubernetes service and endpoints.

A service creates an abstraction around pods or an external service. When an application running in a pod requests the service, that request is routed to the endpoints that share the service name.

Define a service named external-vault and a corresponding endpoint configured to address the EXTERNAL_VAULT_ADDR.

$ cat > external-vault.yaml <<EOF
---
apiVersion: v1
kind: Service
metadata:
  name: external-vault
  namespace: default
spec:
  ports:
  - protocol: TCP
    port: 8200
---
apiVersion: v1
kind: Endpoints
metadata:
  name: external-vault
subsets:
  - addresses:
      - ip: $EXTERNAL_VAULT_ADDR
    ports:
      - port: 8200
EOF

Create the external-vault service.

$ kubectl apply --filename external-vault.yaml

Verify that the external-vault service is addressable from within the devwebapp pod.

$ kubectl exec devwebapp -- curl -s http://external-vault:8200/v1/sys/seal-status | jq

The result displays the status of the Vault server.

{
  "type": "shamir",
  "initialized": true,
  "sealed": false,
  "t": 1,
  "n": 1,
  "progress": 0,
  "nonce": "",
  "version": "1.5.0",
  "migration": false,
  "cluster_name": "vault-cluster-44ba824c",
  "cluster_id": "adc0bb6a-e330-3e7a-e0c7-38061c3bf191",
  "recovery_seal": false,
  "storage_type": "inmem"
}

Create a pod that sets the VAULT_ADDR to the external-vault service.

$ kubectl apply --filename=pod-devwebapp-through-service.yaml
deployment.apps/devwebapp-through-service created

The devwebapp-through-service pod addresses Vault through the service instead of the hard-coded network address.

Get all the pods in the default namespace.

$ kubectl get pods
NAME                        READY   STATUS    RESTARTS   AGE
devwebapp                   1/1     Running   0          36m
devwebapp-through-service   1/1     Running   0          20s

Wait until the devwebapp-through-service pod is running and ready (1/1).

Request content served at localhost:8080 from within the devwebapp-through-service pod.

$ kubectl exec devwebapp-through-service -- curl -s localhost:8080 ; echo

The result displays the secret is defined at the path secret/data/devwebapp/config.

{"password"=>"salsa", "username"=>"giraffe"}

The web application authenticates and requests the secret from the external Vault server that it found through the external-vault service.

»Install the Vault Helm chart configured to address an external Vault

The Vault Helm chart can deploy only the Vault Agent Injector service configured to target an external Vault. The injector service enables the authentication and secret retrieval for the applications, by adding Vault Agent containers as they are written to the pod automatically it includes specific annotations.

In this section, you will install the Vault Helm chart to run only the injector service, configure Vault's Kubernetes authentication, create a role to access a secret, and patch a deployment.

»Install the Vault Helm chart

The Vault Helm chart is able to install only the Vault Agent Injector service.

Add the HashiCorp Helm repository.

$ helm repo add hashicorp https://helm.releases.hashicorp.com
"hashicorp" has been added to your repositories

Update all the repositories to ensure helm is aware of the latest versions.

$ helm repo update
Hang tight while we grab the latest from your chart repositories...
...Successfully got an update from the "hashicorp" chart repository
Update Complete. ⎈Happy Helming!⎈

Install the latest version of the Vault server running in external mode.

$ helm install vault hashicorp/vault \
    --set "injector.externalVaultAddr=http://external-vault:8200"

The injector.externalVaultAddr is assigned the address of the Kubernetes service defined in the Deploy service and endpoints to address an external Vault step.

The Vault Agent Injector pod is deployed in the default namespace.

Get all the pods in the default namespace.

$ kubectl get pods
NAME                                    READY   STATUS    RESTARTS   AGE
devwebapp                               1/1     Running   0          84m
devwebapp-through-service               1/1     Running   0          48m
vault-agent-injector-7b6cd469d8-8svg5   1/1     Running   0          15s

Wait until the vault-agent-injector pod reports that it is running and ready (1/1).

The Helm chart creates the vault service account. The service account secret is necessary to configure Vault's Kubernetes auth method.

Describe the vault service account.

$ kubectl describe serviceaccount vault
Name:                vault
Namespace:           default
Labels:              app.kubernetes.io/instance=vault
                     app.kubernetes.io/managed-by=Helm
                     app.kubernetes.io/name=vault
                     helm.sh/chart=vault-0.8.0
Annotations:         meta.helm.sh/release-name: vault
                     meta.helm.sh/release-namespace: default
Image pull secrets:  <none>
Mountable secrets:   vault-token-7hnch
Tokens:              vault-token-7hnch
Events:              <none>

The name of the mountable secret is displayed.

Create a variable named VAULT_HELM_SECRET_NAME that stores the secret name.

$ VAULT_HELM_SECRET_NAME=$(kubectl get secrets --output=json | jq -r '.items[].metadata | select(.name|startswith("vault-token-")).name')

This command filters the secrets by those that start with vault-token- and returns the name of token.

Describe the vault-token secret.

$ kubectl describe secret $VAULT_HELM_SECRET_NAME

The secret displays its metadata and token value.

»Configure Kubernetes authentication

Vault provides a Kubernetes authentication method that enables clients to authenticate with a Kubernetes Service Account Token.

Enable the Kubernetes authentication method.

$ vault auth enable kubernetes
Success! Enabled kubernetes auth method at: kubernetes/

Vault accepts this service token from any client within the Kubernetes cluster. During authentication, Vault verifies that the service account token is valid by querying a configured Kubernetes endpoint. To configure it correctly requires capturing the JSON web token (JWT) for the service account, the Kubernetes CA certificate, and the Kubernetes host URL.

First, get the JSON web token (JWT) from the secret.

$ TOKEN_REVIEW_JWT=$(kubectl get secret $VAULT_HELM_SECRET_NAME --output='go-template={{ .data.token }}' | base64 --decode)

Next, retrieve the Kubernetes CA certificate.

$ KUBE_CA_CERT=$(kubectl config view --raw --minify --flatten --output='jsonpath={.clusters[].cluster.certificate-authority-data}' | base64 --decode)

Next, retrieve the Kubernetes host URL.

$ KUBE_HOST=$(kubectl config view --raw --minify --flatten --output='jsonpath={.clusters[].cluster.server}')

Finally, configure the Kubernetes authentication method to use the service account token, the location of the Kubernetes host, and its certificate.

$ vault write auth/kubernetes/config \
        token_reviewer_jwt="$TOKEN_REVIEW_JWT" \
        kubernetes_host="$KUBE_HOST" \
        kubernetes_ca_cert="$KUBE_CA_CERT"

For a Vault client to read the secret data defined in the Start Vault section requires that the read capability be granted for the path secret/data/devwebapp/config.

Write out the policy named devwebapp that enables the read capability for secrets at path secret/data/devwebapp/config

$ vault policy write devwebapp - <<EOF
path "secret/data/devwebapp/config" {
  capabilities = ["read"]
}
EOF

Create a Kubernetes authentication role named devweb-app.

$ vault write auth/kubernetes/role/devweb-app \
        bound_service_account_names=internal-app \
        bound_service_account_namespaces=default \
        policies=devwebapp \
        ttl=24h

The role connects the Kubernetes service account, internal-app, and namespace, default, with the Vault policy, devwebapp. The tokens returned after authentication are valid for 24 hours.

»Inject secrets into the pod

The Vault Agent Injector only modifies a pod or deployment if it contains a specific set of annotations.

Display the pod with annotations pod-devwebapp-with-annotations.yaml.

$ cat patch-02-inject-secrets.yml
apiVersion: v1
kind: Pod
metadata:
  name: devwebapp-with-annotations
  labels:
    app: devwebapp-with-annotations
  annotations:
    vault.hashicorp.com/agent-inject: "true"
    vault.hashicorp.com/role: "devweb-app"
    vault.hashicorp.com/agent-inject-secret-credentials.txt: "secret/data/devwebapp/config"
spec:
  serviceAccountName: internal-app
  containers:
    - name: app
      image: burtlo/devwebapp-ruby:k8s
      env:
      - name: VAULT_ADDR
        value: "http://external-vault:8200"
      - name: VAULT_TOKEN
        value: root

These annotations define a partial structure of the deployment schema and are prefixed with vault.hashicorp.com.

  • the agent-inject enables the Vault Agent Injector service
  • the role is the Vault Kubernetes authentication role
  • the agent-inject-secret-FILEPATH prefixes the path of the file, credentials.txt written to the /vault/secrets directory. The value is the path to the secret defined in Vault.

Create the devwebapp-with-annotations pod.

$ kubectl apply --filename pod-devwebapp-with-annotations.yaml
pod/devwebapp-with-annotations created

Get all the pods in the default namespace.

$ kubectl get pods
NAME                                    READY   STATUS    RESTARTS   AGE
devwebapp                               1/1     Running   0          84m
devwebapp-through-service               1/1     Running   0          48m
devwebapp-with-annotations              2/2     Running   0          39s
vault-agent-injector-7b6cd469d8-8svg5   1/1     Running   0          17m

Wait until the devwebapp-with-annotations pod reports that it is running and ready (2/2).

The Vault Agent Injector service created an additional container in the pod that automatically writes the secrets to the app container at the filepath /vault/secrets/credentials.txt.

Display the secrets written to the file /vault/secrets/secret-credentials.txt on the devwebapp-with-annotations pod.

$ kubectl exec -it devwebapp-with-annotations -c app -- cat /vault/secrets/credentials.txt

The result displays the unformatted secret data present on the container.

data: map[password:salsa username:giraffe]
metadata: map[created_time:2021-06-03T15:41:42.676582Z deletion_time: destroyed:false version:1]

The application in this pod still retrieves the secrets directly, but now that the injector service is deployed and capable of retrieving secrets for the application, future updates can remove that application logic.

»Next Steps

You deployed Vault external to a Kubernetes cluster and deployed pods that leveraged it as a secrets store. First, through a hard-coded network address. Second, aliased behind a Kubernetes service and endpoint. And finally, through the Vault Helm's chart and the injector service with annotations applied to a deployment. Learn more about the Vault Helm chart by reading the documentation, exploring the project source code, exploring how pods can retrieve secrets through the Vault Injector service via annotations, or secrets mounted on ephemeral volumes.

Back to Collection
Hashicorp
stdin: is not a tty