Kubernetes

Integrate a Kubernetes Cluster with an External Vault

Application deployments in a Kubernetes cluster can leverage Vault to manage their secrets. Vault run internally is explored in the Vault Installation to Minikube via Helm and Injecting Secrets into Kubernetes Pods via Vault Helm Sidecar guides. There are situations where you may have an existing Vault service that is external to the cluster.

In this guide, you will run Vault locally, start a Kubernetes cluster with Minikube, deploy an application that retrieves secrets from this Vault, and configure an injector only deployment to inject secrets into the pods from this Vault.

Prerequisites

This guide requires the Kubernetes command-line interface (CLI) and the Helm CLI installed, Minikube, the Vault Helm charts, the sample web application, and additional configuration to bring it all together.

This guide was last tested 24 Mar 2020 on a macOS 10.15.3 using this configuration.

Docker version.

$ docker version
Client: Docker Engine - Community
  Version:          19.03.8
  ## ...

Minikube version.

$ minikube version
minikube version: v1.8.2
commit: eb13446e786c9ef70cb0a9f85a633194e62396a1

Helm version.

$ helm version
version.BuildInfo{Version:"v3.1.2", GitCommit:"d878d4d45863e42fd5cff6743294a11d28a9abce", GitTreeState:"clean", GoVersion:"go1.14"}

These are recommended software versions and the output displayed may vary depending on your environment and the software versions you use.

First, follow the directions for installing Minikube, including VirtualBox or similar.

Next, install kubectl CLI and helm CLI.

Install kubectl with Homebrew.

$ brew install kubernetes-cli

Install helm with Homebrew.

$ brew install helm

Next, retrieve the web application and additional configuration by cloning the hashicorp/vault-guides repository from GitHub.

$ git clone https://github.com/hashicorp/vault-guides.git

This repository contains supporting content for all of the Vault learn guides. The content specific to this guide can be found within a sub-directory.

Go into the vault-guides/operations/provision-vault/kubernetes/minikube/external-vault directory.

$ cd vault-guides/operations/provision-vault/kubernetes/minikube/external-vault

Start Vault

Vault running external of a Kubernetes cluster can be addressed by any of its pods as long as the Vault server is network addressable. Running Vault locally alongside of Minikube is possible if the Vault server is bound to the same network as the cluster.

Start a Vault dev server, with root as the root token, and that listens for requests at 0.0.0.0:8200.

$ vault server -dev -dev-root-token-id root -dev-listen-address 0.0.0.0:8200

Setting the -dev-listen-address to 0.0.0.0:8200 overrides the default address of a Vault dev server (127.0.0.1:8200) and enables Vault to be addressable by the Kubernetes cluster and its pods because it binds to a shared network.

Export an environment variable for the vault CLI to address the Vault server.

$ export VAULT_ADDR=http://0.0.0.0:8200

The web application that you deploy, expects Vault to store a username and password stored at the path secret/devwebapp/config. To create this secret requires that a key-value secret engine is enabled and a username and password is put at the specified path. By default the Vault dev server starts with a key-value secrets engine enabled at the path prefixed with secret.

Create a secret at path secret/devwebapp/config with a username and password.

$ vault kv put secret/devwebapp/config username='giraffe' password='salsa'
Key              Value
---              -----
created_time     2020-02-14T07:58:29.973873Z
deletion_time    n/a
destroyed        false
version          1

Verify that the secret is defined at the path secret/data/devwebapp/config.

$ vault read -format json secret/data/devwebapp/config | jq ".data.data"
{
  "password": "salsa",
  "username": "giraffe"
}

The Vault server, with secret, is ready to be addressed by a Kubernetes cluster and the pods deployed in it.

Start Minikube

Minikube is a CLI tool that provisions and manages the lifecycle of single-node Kubernetes clusters locally inside Virtual Machines (VM) on your system.

Start a Kubernetes cluster.

$ minikube start
😄  minikube v1.8.2 on Darwin 10.15.3
✨  Automatically selected the hyperkit driver. Other choices: virtualbox, docker
🔥  Creating hyperkit VM (CPUs=2, Memory=4000MB, Disk=20000MB) ...
🐳  Preparing Kubernetes v1.17.3 on Docker 19.03.6 ...
🚀  Launching Kubernetes ...
🌟  Enabling addons: default-storageclass, storage-provisioner
⌛  Waiting for cluster to come online ...
🏄  Done! kubectl is now configured to use "minikube"

The initialization process takes several minutes as it retrieves any necessary dependencies and executes various container images.

Verify the status of the Minikube cluster.

$ minikube status
host: Running
kubelet: Running
apiserver: Running
kubeconfig: Configured

The host, kubelet, apiserver report that they are running. The kubectl, a command line interface (CLI) for running commands against Kubernetes cluster, is also configured to communicate with this recently started cluster.

Minikube provides a visual representation of the status in a web-based dashboard. This interface makes it easy to view cluster activity and delve into the issues affecting it.

In another terminal, launch the minikube dashboard.

$ minikube dashboard

The operating system's default browser opens and displays the dashboard.

Determine the Vault address

A service bound to all networks on the host, as you configured Vault, is addressable by pods within Minikube's cluster by sending requests to the gateway address of the Kubernetes cluster.

Start a minikube SSH session.

$ minikube ssh
                         _             _
            _         _ ( )           ( )
  ___ ___  (_)  ___  (_)| |/')  _   _ | |_      __
/' _ ` _ `\| |/' _ `\| || , <  ( ) ( )| '_`\  /'__`\
| ( ) ( ) || || ( ) || || |\`\ | (_) || |_) )(  ___/
(_) (_) (_)(_)(_) (_)(_)(_) (_)`\___/'(_,__/'`\____)

Within this SSH session, retrieve the value of the Minikube host.

$ route -n | grep ^0.0.0.0 | awk '{ print $2 }'
192.168.64.1

Next, retrieve the status of the Vault server to verify network connectivity.

$ route -n | grep ^0.0.0.0 | awk '{ print $2 }' | xargs -I{} curl -s http://{}:8200/v1/sys/seal-status | jq
{
  "type": "shamir",
  "initialized": true,
  "sealed": false,
  "t": 1,
  "n": 1,
  "progress": 0,
  "nonce": "",
  "version": "1.3.0",
  "migration": false,
  "cluster_name": "vault-cluster-e719a43c",
  "cluster_id": "e6412343-2873-a319-bc92-e3d592ee9584",
  "recovery_seal": false,
  "storage_type": "inmem"
}

The output displays that Vault is initialized and unsealed. This confirms that pods within your cluster are able to reach Vault given that each pod is configured to use the gateway address.

Next, exit the Minikube SSH session.

$ exit

Finally, create a variable named EXTERNAL_VAULT_ADDR to capture the Minikube gateway address.

$ EXTERNAL_VAULT_ADDR=$(minikube ssh "route -n | grep ^0.0.0.0 | awk '{ print \$2 }'" | tr -d '\r')

Verify that the variable contains the ip address you saw when executed in the minikube shell.

$ echo $EXTERNAL_VAULT_ADDR
192.168.64.1

Deploy application with hard-coded Vault address

The most direct way for a pod within the cluster to address Vault is with a hard-coded network address defined within the application code or provided as an environment variable. We've created and published a web application that you will deploy with the Vault address overriden.

First, create a Kubernetes service account for the pods to use to authenticate.

$ cat <<EOF | kubectl apply -f -
apiVersion: v1
kind: ServiceAccount
metadata:
  name: internal-app
EOF

Create a deployment with this web application that sets the VAULT_ADDR to EXTERNAL_VAULT_ADDR.

$ cat <<EOF | kubectl apply -f -
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: devwebapp
  labels:
    app: devwebapp
spec:
  replicas: 1
  selector:
    matchLabels:
      app: devwebapp
  template:
    metadata:
      labels:
        app: devwebapp
    spec:
      serviceAccountName: internal-app
      containers:
      - name: app
        image: burtlo/devwebapp-ruby:k8s
        imagePullPolicy: Always
        env:
        - name: VAULT_ADDR
          value: "http://$EXTERNAL_VAULT_ADDR:8200"
EOF

The web application, targeting the external Vault, is deployed as a pod within the default namespace.

Get all the pods within the default namespace.

$ kubectl get pods
NAME                         READY   STATUS    RESTARTS   AGE
devwebapp-68cc55948b-w9745   1/1     Running   0          4m

Request the web application display its content.

$ kubectl exec devwebapp-68cc55948b-w9745 -- curl -s localhost:8080
{"password"=>"salsa", "username"=>"giraffe"}%

The web application authenticates with the Vault server using the root token and returns the secret defined at the path secret/data/devwebapp/config. This hard-coded approach is an effective solution if the address to the Vault server does not change.

Deploy service and endpoints to address an external Vault

An external Vault may not have a static network address that services within the cluster can rely upon. When Vault's network address changes each service also needs to change to continue its operation. Another approach to manage this network address is to define a Kubernetes service and endpoints.

A service creates an abstraction around pods or an external service. When an application running in a pod requests the service, that request is routed to the endpoints that share the service name.

Deploy a service named external-vault and a corresponding endpoint configured to address the EXTERNAL_VAULT_ADDR.

$ cat <<EOF | kubectl apply -f -
---
apiVersion: v1
kind: Service
metadata:
  name: external-vault
  namespace: default
spec:
  ports:
  - protocol: TCP
    port: 8200
---
apiVersion: v1
kind: Endpoints
metadata:
  name: external-vault
subsets:
  - addresses:
      - ip: $EXTERNAL_VAULT_ADDR
    ports:
      - port: 8200
EOF

Verify that the external-vault service is addressable from the existing pod.

$ kubectl exec devwebapp-68cc55948b-w9745 -- curl -s http://external-vault:8200/v1/sys/seal-status | jq
{
  "type": "shamir",
  "initialized": true,
  "sealed": false,
  "t": 1,
  "n": 1,
  "progress": 0,
  "nonce": "",
  "version": "1.3.0",
  "migration": false,
  "cluster_name": "vault-cluster-e719a43c",
  "cluster_id": "e6412343-2873-a319-bc92-e3d592ee9584",
  "recovery_seal": false,
  "storage_type": "inmem"
}

Next, create a deployment that sets the VAULT_ADDR to the external-vault service.

$ kubectl apply -f deployment-01-external-vault-service.yml
deployment.apps/devwebapp-through-service created

This deployment named devwebapp-through-service creates a pod that addresses Vault through the service instead of the the hard-coded network address.

To verify, get all the pods in the default namespace.

$ kubectl get pods
NAME                                        READY   STATUS    RESTARTS   AGE
devwebapp-54b89c546b-zd8np                  1/1     Running   0          36m
devwebapp-through-service-6b4b79994-9t7v7   1/1     Running   0          20s

The pod name appears here with the prefix devwebapp-through-service.

Finally, request the web application display its content.

$ kubectl exec devwebapp-through-service-6b4b79994-9t7v7 -- curl -s localhost:8080
{"password"=>"salsa", "username"=>"giraffe"}%

Install the Vault Helm chart configured to address an external Vault

The Vault Helm chart can deploy only the Vault Agent injector service configured to target an external Vault. The injector service enables the authentication and secret retrieval for the applications, by adding Vault Agent containers as they are written to the pod automatically when a deployment includes specific annotations.

In this section, you will create a Kubernetes service account; configure Vault's Kubernetes authentication and create a role to access a secret; install the Vault Helm chart to run only the injector service; and patch a deployment.

Define a Kubernetes service account

Create a service account with the neccessary permissions to allow Vault to perform token reviews with Kubernetes.

$ cat <<EOF | kubectl create -f -
---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: vault-auth
---
apiVersion: v1
kind: Secret
metadata:
  name: vault-auth
  annotations:
    kubernetes.io/service-account.name: vault-auth
type: kubernetes.io/service-account-token
---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRoleBinding
metadata:
  name: role-tokenreview-binding
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: system:auth-delegator
subjects:
  - kind: ServiceAccount
    name: vault-auth
    namespace: default
EOF
serviceaccount/vault-auth created
secret/vault-auth created
clusterrolebinding.rbac.authorization.k8s.io/role-tokenreview-binding created

Configure Kubernetes authentication

Vault provides a Kubernetes authentication method that enables clients to authenticate with a Kubernetes Service Account Token.

Enable the Kubernetes authentication method.

$ vault auth enable kubernetes
Success! Enabled kubernetes auth method at: kubernetes/

Vault accepts this service token from any client within the Kubernetes cluster. During authentication, Vault verifies that the service account token is valid by querying a configured Kubernetes endpoint. To configure it correctly requires capturing the JSON web token (JWT) for the service account, the Kubernetes CA certificate, and the Kubernetes host URL.

First, get the JSON web token (JWT) for this service account.

$ TOKEN_REVIEW_JWT=$(kubectl get secret vault-auth -o go-template='{{ .data.token }}' | base64 --decode)

Next, retrieve the Kubernetes CA certificate.

$ KUBE_CA_CERT=$(kubectl config view --raw --minify --flatten -o jsonpath='{.clusters[].cluster.certificate-authority-data}' | base64 --decode)

Next, retrieve the Kubernetes host URL.

$ KUBE_HOST=$(kubectl config view --raw --minify --flatten -o jsonpath='{.clusters[].cluster.server}')

Finally, configure the Kubernetes authentication method to use the service account token, the location of the Kubernetes host, and its certificate.

$ vault write auth/kubernetes/config \
        token_reviewer_jwt="$TOKEN_REVIEW_JWT" \
        kubernetes_host="$KUBE_HOST" \
        kubernetes_ca_cert="$KUBE_CA_CERT"
Success! Data written to: auth/kubernetes/config

For a Vault client to read the secret data defined in the previous step requires that the read capability be granted for the path secret/data/devwebapp/config.

Write out the policy named devweb-app that enables the read capability for secrets at path secret/data/devwebapp/config

$ vault policy write devweb-app - <<EOF
path "secret/data/devwebapp/config" {
  capabilities = ["read"]
}
EOF
Success! Uploaded policy: devweb-app

Create a Kubernetes authentication role named devweb-app.

$ vault write auth/kubernetes/role/devweb-app \
        bound_service_account_names=internal-app \
        bound_service_account_namespaces=default \
        policies=devweb-app \
        ttl=24h
Success! Data written to: auth/kubernetes/role/devweb-app

The role connects the Kubernetes service account, internal-app, and namespace, default, with the Vault policy, devweb-app. The tokens returned after authentication are valid for 24 hours.

Install the Vault Helm chart

The Vault Helm chart is able to install only the Vault Agent injector service.

Install the Vault Helm chart version 0.4.0, setting the injector.externalVaultAddr to use the external-vault service.

$ helm install vault \
    --set "injector.externalVaultAddr=http://external-vault:8200" \
    https://github.com/hashicorp/vault-helm/archive/v0.4.0.tar.gz

NAME: vault
## ...

To verify, get all the pods in the default namespace.

$ kubectl get pods
NAME                                        READY   STATUS    RESTARTS   AGE
devwebapp-54b89c546b-zd8np                  1/1     Running   0          84m
devwebapp-through-service-6b4b79994-9t7v7   1/1     Running   0          48m
vault-agent-injector-7b6cd469d8-8svg5       1/1     Running   0          15s

Inject secrets into the pod

The Vault Agent injector only modifies a deployment if it contains a specific set of annotations. An existing deployment may have its definition patched to include the necessary annotations.

View the deployment patch patch-02-inject-secrets.yml.

$ cat patch-02-inject-secrets.yml
spec:
  template:
    metadata:
      annotations:
        vault.hashicorp.com/agent-inject: "true"
        vault.hashicorp.com/role: "devweb-app"
        vault.hashicorp.com/agent-inject-secret-credentials.txt: "secret/data/devwebapp/config"

These annotations define a partial structure of the deployment schema and are prefixed with vault.hashicorp.com.

  • agent-inject enables the Vault Agent injector service
  • role is the Vault Kubernetes authentication role
  • agent-inject-secret-FILEPATH prefixes the path of the file, credentials.txt written to the /vault/secrets directory. The value is the path to the secret defined in Vault.

Patch the existing devwebapp deployment with the annoations to write the secrets to the pod.

$ kubectl patch deployment devwebapp --patch "$(cat patch-02-inject-secrets.yml)"
deployment.apps/devwebapp patched

The application pod is redeployed and the secrets are now available on the pod at the filepath /vault/secrets/credentials.txt.

Finally, Verify that the secrets are written to the file /vault/secrets/secret-credentials.txt.

$ kubectl exec -it devwebapp-54b89c546b-zd8np -c app -- cat /vault/secrets/credentials.txt
data: map[password:salsa username:giraffe]
metadata: map[created_time:2019-12-20T18:17:50.930264759Z deletion_time: destroyed:false version:2]

The unformatted secret data is present on the container.

Next Steps

You deployed Vault external to a Kubernetes cluster and deployed pods that leveraged it as a secrets store. First, through a hard-coded network address. Second, aliased behind a Kubernetes service and endpoint. And finally, through the Vault Helm's chart and the injector service with annotations applied to a deployment. Learn more about the Vault Helm chart by reading the documentation, exploring the project source code, exploring how pods can retrieve secrets through the Vault Injector service via annotations, or secrets mounted on ephemeral volumes.