Run Vault on Kubernetes

Injecting Secrets into Kubernetes Pods via Vault Helm Sidecar

Deplying applications that act as secret consumers of Vault require the application to:

  • Authenticate and acquire a client token.
  • Manage the lifecycle of the token.
  • Retrieve secrets from Vault.
  • Manage the leases of any dynamic secrets.

Running Vault Agent along with your applications enable your applications to offload this workflow to it while remaining unaware of Vault. However, this does mean that with every application you deployment you also need to this daemon process.

The Vault Helm chart enables you to run Vault and the Vault Agent injector service. This injector service leverages the Kubernetes mutating admission webhook to intercept pods that define specific annotations and inject a Vault Agent container to manage these secrets. This is beneficial because:

  • Applications remain Vault unaware as the secrets are stored on the file-system within their container.
  • Existing deployments require no change; as annotations can be patched.
  • Access to secrets can be enforced via Kubernetes service accounts and namespaces

In this guide, you setup Vault and this injector service with the Vault Helm chart. Then deploy several applications to demonstrate how this new injector service retrieves and writes these secrets for the applications use.

Prerequisites

This guide requires the Kubernetes command-line interface (CLI) and the Helm CLI installed, Minikube, and additional configuration to bring it all together.

This guide was last tested 20 Dec 2019 on a macOS 10.15.2 using this configuration:

$ docker version
Client: Docker Engine - Community
 Version:           19.03.5
 ...

$ minikube version
minikube version: v1.5.2
commit: 792dbf92a1de583fcee76f8791cff12e0c9440ad

$ helm version
Client: &version.Version{SemVer:"v2.16.1", GitCommit:"bbdfe5e7803a12bbdf97e94cd847859890cf4050", GitTreeState:"clean"}
# Tiller's version appears after `helm init` later in the guide.
Error: could not find tiller

Although we recommend these software versions, the output you see may vary depending on your environment and the software versions you use.

First, follow the directions for installing Minikube, including VirtualBox or similar.

Next, install kubectl CLI and helm CLI.

On Mac with Homebrew.

$ brew install kubernetes-cli
$ brew install helm@2

On Windows with Chocolatey:

$ choco install kubernetes-cli
$ choco install kubernetes-helm --version 2.16.0

Next, retrieve the web application and additional configuration by cloning the hashicorp/vault-guides repository from GitHub.

$ git clone https://github.com/hashicorp/vault-guides.git

This repository contains supporting content for all of the Vault learn guides. The content specific to this guide can be found within a sub-directory.

Go into the vault-guides/operations/provision-vault/kubernetes/minikube/vault-agent-sidecar directory.

$ cd vault-guides/operations/provision-vault/kubernetes/minikube/vault-agent-sidecar

Start Minikube

Minikube is a CLI tool that provisions and manages the lifecycle of single-node Kubernetes clusters locally inside Virtual Machines (VM) on your system.

Start a Kubernetes cluster with 4096 Megabytes (MB) of memory:

$ minikube start --memory 4096
😄  minikube v1.5.2 on Darwin 10.15.2
✨  Automatically selected the 'hyperkit' driver (alternates: [virtualbox])
🔥  Creating hyperkit VM (CPUs=2, Memory=4096MB, Disk=20000MB) ...
🐳  Preparing Kubernetes v1.16.2 on Docker '18.09.9' ...
🚜  Pulling images ...
🚀  Launching Kubernetes ...
⌛  Waiting for: apiserver
🏄  Done! kubectl is now configured to use "minikube"

The --memory is set to 4096 MB to ensure there is plenty of memory for all the resources to run concurrently. The initialization process takes several minutes as it retrieves any necessary dependencies and executes various container images.

Verify the status of the Minikube cluster:

$ minikube status
host: Running
kubelet: Running
apiserver: Running
kubeconfig: Configured

The host, kubelet, apiserver report that they are running. The kubectl, a command line interface (CLI) for running commands against Kubernetes cluster, is also configured to communicate with this recently started cluster.

Minikube provides a visual representation of the status in a web-based dashboard. This interface makes it easy to view cluster activity and delve into the issues affecting it.

In another terminal, launch the minikube dashboard:

$ minikube dashboard

The operating system's default browser opens and displays the dashboard.

Minikube Dashboard

Initialize Helm

Helm is a package manager that provides the community the ability to share charts that describe the operation of resources operating within Kubernetes. Helm requires initialization so that it can load an in-cluster component called Tiller. Tiller interacts directly with the Kubernetes API Server to install, upgrade, query, and remove Kubernetes resources.

Initialize Helm and start Tiller:

$ helm init
$HELM_HOME has been configured at $HOME/.helm.

Tiller (the Helm server-side component) has been installed into your Kubernetes Cluster.

Please note: by default, Tiller is deployed with an insecure 'allow unauthenticated users' policy.
To prevent this, run `helm init` with the --tiller-tls-verify flag.
For more information on securing your installation see: https://docs.helm.sh/using_helm/#securing-your-helm-installation

Tiller runs as a pod within the kube-system namespace. A pod is a Kubernetes resource that represents one or more cooperating processes that form a cohesive unit of service. Namespaces, another Kubernetes resource, provide a scope for names. The kube-system is the namespace specifically for resources created to support the Kubernetes system.

Verify that Tiller is running by getting all the pods within the kube-system namespace:

$ kubectl get pods --namespace kube-system
NAME                                    READY   STATUS    RESTARTS   AGE
coredns-5c98db65d4-s8cdv                1/1     Running   1          7m17s
coredns-5c98db65d4-vh5tw                1/1     Running   1          7m17s
etcd-minikube                           1/1     Running   0          6m20s
kube-addon-manager-minikube             1/1     Running   0          6m19s
kube-apiserver-minikube                 1/1     Running   0          6m12s
kube-controller-manager-minikube        1/1     Running   0          6m9s
kube-proxy-llgmm                        1/1     Running   0          7m17s
kube-scheduler-minikube                 1/1     Running   0          6m12s
kubernetes-dashboard-7b8ddcb5d6-7gs2l   1/1     Running   0          7m16s
storage-provisioner                     1/1     Running   0          7m16s
tiller-deploy-75f6c87b87-n4db8          1/1     Running   0          21s

The Tiller service appears here as the pod named tiller-deploy-75f6c87b87-n4db8. It reports that it is ready and running.

Verify that Tiller is running, through the dashboard, by getting all the pods within the kube-system namespace:

Minikube dashboard highlighting the "kube-system" namespace

Install the Vault Helm chart

The recommended way to run Vault on Kubernetes is via the Helm chart. This installs and configures all the necessary components to run Vault in several different modes.

Install the Vault Helm chart version 0.3.0 with pods prefixed with the name vault:

$ helm install --name vault \
    --set "server.dev.enabled=true" \
    https://github.com/hashicorp/vault-helm/archive/v0.3.0.tar.gz
NAME:   vault
LAST DEPLOYED: Fri Dec 20 11:56:33 2019
NAMESPACE: default
STATUS: DEPLOYED

RESOURCES:

...

NOTES:

...

Your release is named vault. To learn more about the release, try:

  $ helm status vault
  $ helm get vault

To verify, get all the pods within the default namespace:

$ kubectl get pods
NAME                                    READY   STATUS    RESTARTS   AGE
vault-0                                 1/1     Running   0          80s
vault-agent-injector-5945fb98b5-tpglz   1/1     Running   0          80s

The Helm chart creates a Vault server pod and Vault-Agent injector pod.

The vault-0 pod starts as a Vault service in development mode. The vault-agent-injector-5945fb98b5-tpglz pod performs the injection based on the annotations present or patched on a deployment.

Set a secret in Vault

The applications that you deploy in the later steps expect Vault to store a username and password stored at the path internal/database/config. To create this secret requires that a key-value secret engine is enabled and a username and password is put at the specified path.

Start an interactive shell session on the vault-0 pod:

$ kubectl exec -it vault-0 /bin/sh
/ $

Your system prompt is replaced with a new prompt / $. Commands issued at this prompt are executed on the vault-0 container.

Enable kv-v2 secrets at the path internal:

/ $ vault secrets enable -path=internal kv-v2
Success! Enabled the kv-v2 secrets engine at: internal/

Put a username and password secret at the path internal/exampleapp/config:

$ vault kv put internal/database/config username="db-readonly-username" password="db-secret-password"
Key              Value
---              -----
created_time     2019-12-20T18:17:01.719862753Z
deletion_time    n/a
destroyed        false
version          1

Verify that the secret is defined at the path internal/database/config:

$ vault kv get internal/database/config
====== Metadata ======
Key              Value
---              -----
created_time     2019-12-20T18:17:50.930264759Z
deletion_time    n/a
destroyed        false
version          1

====== Data ======
Key         Value
---         -----
password    db-secret-password
username    db-readonly-username

The secret is ready for the application.

Configure Kubernetes authentication

Vault provides a Kubernetes authentication method that enables clients to authenticate with a Kubernetes Service Account Token.

Enable the Kubernetes authentication method:

/ $ vault auth enable kubernetes
Success! Enabled kubernetes auth method at: kubernetes/

Vault accepts this service token from any client within the Kubernetes cluster. During authentication, Vault verifies that the service account token is valid by querying a configured Kubernetes endpoint.

Configure the Kubernetes authentication method to use the service account token, the location of the Kubernetes host, and its certificate:

/ $ vault write auth/kubernetes/config \
        token_reviewer_jwt="$(cat /var/run/secrets/kubernetes.io/serviceaccount/token)" \
        kubernetes_host="https://$KUBERNETES_PORT_443_TCP_ADDR:443" \
        kubernetes_ca_cert=@/var/run/secrets/kubernetes.io/serviceaccount/ca.crt
Success! Data written to: auth/kubernetes/config

The token_reviewer_jwt and kubernetes_ca_cert reference files written to the container by Kubernetes. The environment variable KUBERNETES_PORT_443_TCP_ADDR references the internal network address of the Kubernetes host.

For a client to read the secret data defined in the previous step, at internal/database/config, requires that the read capability be granted for the path internal/data/database/config.

Write out the policy named internal-app that enables the read capability for secrets at path internal/data/database/config

/ $ vault policy write internal-app - <<EOH
path "internal/data/database/config" {
  capabilities = ["read"]
}
EOH
Success! Uploaded policy: internal-app

Create a Kubernetes authentication role named internal-app:

/ $ vault write auth/kubernetes/role/internal-app \
        bound_service_account_names=internal-app \
        bound_service_account_namespaces=default \
        policies=internal-app \
        ttl=24h
Success! Data written to: auth/kubernetes/role/internal-app

The role connects the Kubernetes service account, internal-app, and namespace, default, with the Vault policy, internal-app. The tokens returned after authentication are valid for 24 hours.

Lastly, exit the the vault-0 pod:

/ $ exit
$

Define a Kubernetes service account

The Vault Kubernetes authentication role defined a Kubernetes service account named internal-app. This service acount does not yet exist.

Verify that the Kubernetes service account named internal-app does not exist:

$ kubectl get serviceaccounts
NAME                   SECRETS   AGE
default                1         43m
vault                  1         34m
vault-agent-injector   1         34m

This account does not exist but it is necessary for authentication.

View the service account defined in exampleapp-service-account.yml:

$ cat service-account-internal-app.yml
apiVersion: v1
kind: ServiceAccount
metadata:
  name: internal-app

This definition of the service account creates the account with the name internal-app.

Apply the service account definition to create it:

$ kubectl apply --filename service-account-internal-app.yml
serviceaccount/internal-app created

Verify that the service account has been created:

$ kubectl get serviceaccounts
NAME                   SECRETS   AGE
default                1         52m
internal-app           1         13s
vault                  1         43m
vault-agent-injector   1         43m

The name of the service account here aligns with the name assigned to the bound_service_account_names field when creating the internal-app role when configuring the Kubernetes authentication.

Launch an application

We've created a sample application, published it to DockerHub, and created a Kubernetes deployment that launches this appliation.

View the deployment for the orgchart application:

$ cat deployment-01-orgchart.yml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: orgchart
  labels:
    app: vault-agent-injector-demo
spec:
  selector:
    matchLabels:
      app: vault-agent-injector-demo
  replicas: 1
  template:
    metadata:
      annotations:
      labels:
        app: vault-agent-injector-demo
    spec:
      serviceAccountName: internal-app
      containers:
        - name: orgchart
          image: jweissig/app:0.0.1

The name of this deployment is orgchart. The spec.template.spec.serviceAccountName defines the service account internal-app to run this container under.

Apply the deployment defined in deployment-01-orgchart.yml:

$ kubectl apply --filename deployment-01-orgchart.yml
deployment.apps/orgchart created

The application runs as a pod within the default namespace.

Get all the pods within the default namespace:

$ kubectl get pods
NAME                                    READY   STATUS    RESTARTS   AGE
orgchart-69697d9598-l878s               1/1     Running   0          18s
vault-0                                 1/1     Running   0          58m
vault-agent-injector-5945fb98b5-tpglz   1/1     Running   0          58m

The orgchart deployment appears here as the pod named orgchart-69697d9598-l878s.

The Vault-Agent injector looks for deployments that define specific annotations. None of these annotations exist within the current deployment. This means that no secrets are present on the orgchart container within the orgchart-69697d9598-l878s pod.

Verify that no secrets are written to the orgchart container in the orgchart-69697d9598-l878s pod:

$ kubectl exec orgchart-69697d9598-l878s --container orgchart -- ls /vault/secrets
ls: /vault/secrets: No such file or directory
command terminated with exit code 1

Inject secrets into the pod

The deployment is running the pod with the internal-app Kubernetes service account in the default namespace. The Vault Agent injector only modifies a deployment if it contains a very specific set of annotations. A existing deployment may have its definition patched to include the necessary annotations.

View the deployment patch deployment-02-inject-secrets.yml:

$ cat deployment-02-inject-secrets.yml
spec:
  template:
    metadata:
      annotations:
        vault.hashicorp.com/agent-inject: "true"
        vault.hashicorp.com/role: "internal-app"
        vault.hashicorp.com/agent-inject-secret-database-config.txt: "internal/data/database/config"

These annotations define a partial structure of the deployment schem and are prefixed with vault.hashicorp.com.

  • agent-inject enables the Vault Agent injector service
  • role is the Vault Kubernetes authentication role
  • role is the Vault role created that maps back to the K8s serviceaccount
  • agent-inject-secret-FIlEPATH prefixes the path of the file, database-config.txt written to /vault/secrets. The values is the path to the secret defined in Vault.

Patch the orgchart deployment defined in deployment-02-inject-secrets.yml:

$ kubectl patch deployment orgchart --patch "$(cat deployment-02-inject-secrets.yml)"
deployment.apps/orgchart patched

Get all the pods within the default namespace:

$ kubectl get pods
NAME                                    READY   STATUS     RESTARTS   AGE
orgchart-599cb74d9c-s8hhm               0/2     Init:0/1   0          23s
orgchart-69697d9598-l878s               1/1     Running    0          20m
vault-0                                 1/1     Running    0          78m
vault-agent-injector-5945fb98b5-tpglz   1/1     Running    0          78m

A new orgchart deployment starts alongside the existing deployment. When it ready the original terminates and removes itself from the list of active pods. The redeployment is complete when the pod reports READY 2/2.

This new pod now launches two containers. The application container, named orgchart, and the Vault Agent container, named vault-agent.

View the logs of the vault-agent container in the orgchart-599cb74d9c-s8hhm pod:

$ kubectl logs orgchart-599cb74d9c-s8hhm --container vault-agent
==> Vault server started! Log data will stream in below:

==> Vault agent configuration:

                     Cgo: disabled
               Log Level: info
                 Version: Vault v1.3.1

2019-12-20T19:52:36.658Z [INFO]  sink.file: creating file sink
2019-12-20T19:52:36.659Z [INFO]  sink.file: file sink configured: path=/home/vault/.token mode=-rw-r-----
2019-12-20T19:52:36.659Z [INFO]  template.server: starting template server
2019/12/20 19:52:36.659812 [INFO] (runner) creating new runner (dry: false, once: false)
2019/12/20 19:52:36.660237 [INFO] (runner) creating watcher
2019-12-20T19:52:36.660Z [INFO]  auth.handler: starting auth handler
2019-12-20T19:52:36.660Z [INFO]  auth.handler: authenticating
2019-12-20T19:52:36.660Z [INFO]  sink.server: starting sink server
2019-12-20T19:52:36.679Z [INFO]  auth.handler: authentication successful, sending token to sinks
2019-12-20T19:52:36.680Z [INFO]  auth.handler: starting renewal process
2019-12-20T19:52:36.681Z [INFO]  sink.file: token written: path=/home/vault/.token
2019-12-20T19:52:36.681Z [INFO]  template.server: template server received new token
2019/12/20 19:52:36.681133 [INFO] (runner) stopping
2019/12/20 19:52:36.681160 [INFO] (runner) creating new runner (dry: false, once: false)
2019/12/20 19:52:36.681285 [INFO] (runner) creating watcher
2019/12/20 19:52:36.681342 [INFO] (runner) starting
2019-12-20T19:52:36.692Z [INFO]  auth.handler: renewed auth token

Vault Agent manages the token lifecycle and the secret retrieval. The secret is rendered in the orgchart container at the path /vault/secrets/database-config.txt.

Finally, view the secret written to the orgchart container:

$ kubectl exec orgchart-599cb74d9c-s8hhm --container orgchart -- cat /vault/secrets/database-config.txt
data: map[password:db-secret-password username:db-readonly-user]
metadata: map[created_time:2019-12-20T18:17:50.930264759Z deletion_time: destroyed:false version:2]

The secret is present on the container. However, the structure is not in one expected by the application.

Apply a template to the injected secrets

The structure of the injected secrets may need to be structured in a way for the application to use. Before writing the secrets to the file system a template can structure the data. To apply this template a new set of annotations need to be applied.

View the annotations file that contains a template definition:

$ cat deployment-03-inject-secrets-as-template.yml
spec:
  template:
    metadata:
      annotations:
        vault.hashicorp.com/agent-inject: "true"
        vault.hashicorp.com/agent-inject-status: "update"
        vault.hashicorp.com/role: "internal-app"
        vault.hashicorp.com/agent-inject-secret-database-config.txt: "internal/data/database/config"
        vault.hashicorp.com/agent-inject-template-database-config.txt: |
          {{- with secret "internal/data/database/config" -}}
          postgresql://{{ .Data.data.username }}:{{ .Data.data.password }}@postgres:5432/wizard
          {{- end -}}

This patch contains two new annotations:

The template writes the username and password found in the secret to a postgresql connection string.

Apply the updated annotations:

$ kubectl patch deployment orgchart --patch "$(cat deployment-03-inject-secrets-as-template.yml)"
deployment.apps/exampleapp patched

Get all the pods within the default namespace:

$ kubectl get pods
NAME                                    READY   STATUS    RESTARTS   AGE
orgchart-554db4579d-w6565               2/2     Running   0          16s
vault-0                                 1/1     Running   0          126m
vault-agent-injector-5945fb98b5-tpglz   1/1     Running   0          126m

Finally, view the template written to the orgchart container:

$ kubectl exec -it orgchart-554db4579d-w6565 -c orgchart -- cat /vault/secrets/database-config.txt
postgresql://db-readonly-user:db-secret-password@postgres:5432/wizard

Deployment with annotations

Maintaining the annotations as a patch enables you to apply them to any deployment. They may also be coupled with the deployment so that the secrets are inject when the pod is initially deployed.

View the deployment for the payrole application:

$ cat deployment-04-payrole.yml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: payrole
  labels:
    app: vault-agent-injector-demo
spec:
  selector:
    matchLabels:
      app: vault-agent-injector-demo
  replicas: 1
  template:
    metadata:
      annotations:
        vault.hashicorp.com/agent-inject: "true"
        vault.hashicorp.com/agent-inject-status: "update"
        vault.hashicorp.com/role: "internal-app"
        vault.hashicorp.com/agent-inject-secret-database-config.txt: "internal/data/database/config"
        vault.hashicorp.com/agent-inject-template-database-config.txt: |
          {{- with secret "internal/data/database/config" -}}
          postgresql://{{ .Data.data.username }}:{{ .Data.data.password }}@postgres:5432/wizard
          {{- end -}}
      labels:
        app: vault-agent-injector-demo
    spec:
      serviceAccountName: internal-app
      containers:
        - name: payrole
          image: jweissig/app:0.0.1

Apply the deployment defined in deployment-04-payrole.yml:

$ kubectl apply --filename deployment-04-payrole.yml
deployment.apps/payrole created

Get all the pods within the default namespace:

$ kubectl get pods
NAME                                    READY   STATUS    RESTARTS   AGE
orgchart-554db4579d-w6565               2/2     Running   0          29m
payrole-7dc758dc7b-9dc6t                2/2     Running   0          12s
vault-0                                 1/1     Running   0          155m
vault-agent-injector-5945fb98b5-tpglz   1/1     Running   0          155m

Finally, view the template rendered to the payrole container:

$ kubectl exec payrole-7dc758dc7b-9dc6t --container payrole -- cat /vault/secrets/database-config.txt
postgresql://db-readonly-user:db-secret-password@postgres:5432/wizard

Secrets are bound to the service account

Attempts to run a pod with a different service account than the ones listed in the authentication are not be able to access the secrets defined at that path.

View the deployment and service account for the website application:

$ cat deployment-05-website.yml

Apply the deployment and service account defined in deployment-05-website.yml:

$ kubectl apply --filename deployment-05-website.yml
deployment.apps/website created
serviceaccount/external-app created

Get all the pods within the default namespace:

$ kubectl get pods
NAME                                    READY   STATUS     RESTARTS   AGE
orgchart-554db4579d-w6565               2/2     Running    0          29m
payrole-7dc758dc7b-9dc6t                2/2     Running    0          12s
vault-0                                 1/1     Running    0          155m
vault-agent-injector-5945fb98b5-tpglz   1/1     Running    0          155m
website-7fc8b69645-527rf                0/2     Init:0/1   0          76s

The website deployment creates a pod but it does not ever become ready.

View the logs of the vault-agent-init container in the website-7fc8b69645-527rf pod:

$ kubectl logs website-7fc8b69645-527rf --container vault-agent-init
...
2019-12-20T21:36:32.825Z [INFO]  auth.handler: authenticating
2019-12-20T21:36:32.830Z [ERROR] auth.handler: error authenticating: error="Error making API request.

URL: PUT http://vault.default.svc:8200/v1/auth/kubernetes/login
Code: 500. Errors:

* service account name not authorized" backoff=1.562132589

The initialization process is failing because the service account name is not authorized. The service account, external-app is not assigned to any Vault Kubernetes authentication role preventing the initialization to complete.

Secrets are bound to the namespace

Similar to how the secrets are bound to a service account they are also bound to a namespace.

Create the offsite namespace:

$ kubectl create namespace offsite
namespace/offsite created

Set the current context to the offsite namespace:

$ kubectl config set-context --current --namespace offsite
Context "minikube" modified.

Apply the deployment and creat the service account defined in deployment-06-issues.yml:

$ kubectl apply --filename deployment-06-issues.yml
deployment.apps/issues created
serviceaccount/internal-app created

Get all the pods within the offsite namespace:

$ kubectl get pods
NAME                      READY   STATUS     RESTARTS   AGE
issues-7956fff46d-9kzv6   0/2     Init:0/1   0          40s

The issues deployment creates a pod but it does not ever become ready.

View the logs of the vault-agent-init container in the issues-7956fff46d-9kzv6 pod:

$ kubectl logs issues-7956fff46d-9kzv6 --container vault-agent-init
...
2019-12-20T21:43:41.293Z [INFO]  auth.handler: authenticating
2019-12-20T21:43:41.296Z [ERROR] auth.handler: error authenticating: error="Error making API request.

URL: PUT http://vault.default.svc:8200/v1/auth/kubernetes/login
Code: 500. Errors:

* namespace not authorized" backoff=1.9882590740000001

The initialization process is failing because the namespace is not authorized. The namespace, offsite is not assigned to any Vault Kubernetes authentication role preventing the initialization to complete.

Summary and reference

The Vault Helm chart's injector service coupled with deployment annotations make it possible to create applications that are able to leverage Vault as a secrets repository without having to modify application code.

Help and reference

Learn more about this by reading the blog post announcing the "Injecting Vault Secrets into Kubernetes Pods via a Sidecar" blog post or the documentation for Agent Sidecar Injector