Skip to main content
Just announced HashiConf Global full schedule: keynotes, sessions, labs & more. Register Now
Type '/' to Search

A new platform for documentation and tutorials is launching soon.

We are migrating Learn content into HashiCorp Developer, our new developer experience.

Injecting Secrets into Kubernetes Pods via Vault Agent Containers

Deploying applications that act as secret consumers of Vault require the application to:

  • Authenticate and acquire a client token.
  • Manage the lifecycle of the token.
  • Retrieve secrets from Vault.
  • Manage the leases of any dynamic secrets.

Vault Agent takes responsibility for these tasks and enables your applications to remain unaware of Vault. However, this introduces a new requirement that deployments install and configure Vault Agent alongside the application as a sidecar.

The Vault Helm chart enables you to run Vault and the Vault Agent Injector service. This injector service leverages the Kubernetes mutating admission webhook to intercept pods that define specific annotations and inject a Vault Agent container to manage these secrets. This is beneficial because:

  • Applications remain Vault unaware as the secrets are stored on the file-system in their container.
  • Existing deployments require no change; as annotations can be patched.
  • Access to secrets can be enforced via Kubernetes service accounts and namespaces

In this tutorial, you setup Vault and this injector service with the Vault Helm chart. Then you will deploy several applications to demonstrate how this new injector service retrieves and writes these secrets for the applications to use.

»Prerequisites

This tutorial requires the Kubernetes command-line interface (CLI) and the Helm CLI installed, Minikube, and additional configuration to bring it all together.

This tutorial was last tested 23 Apr 2021 on a macOS 11.2.3 using this configuration.

Docker version.

$ docker version
Client: Docker Engine - Community
 Cloud integration: 1.0.12
 Version:           20.10.5
 ## ...

$ docker version
Client: Docker Engine - Community
 Cloud integration: 1.0.12
 Version:           20.10.5
 ## ...

Minikube version.

$ minikube version
minikube version: v1.19.0
commit: 15cede53bdc5fe242228853e737333b09d4336b5

$ minikube version
minikube version: v1.19.0
commit: 15cede53bdc5fe242228853e737333b09d4336b5

Helm version.

$ helm version
version.BuildInfo{Version:"v3.5.4", GitCommit:"1b5edb69df3d3a08df77c9902dc17af864ff05d1", GitTreeState:"dirty", GoVersion:"go1.16.3"}

$ helm version
version.BuildInfo{Version:"v3.5.4", GitCommit:"1b5edb69df3d3a08df77c9902dc17af864ff05d1", GitTreeState:"dirty", GoVersion:"go1.16.3"}

These are recommended software versions and the output displayed may vary depending on your environment and the software versions you use.

First, follow the directions to install Minikube, including VirtualBox or similar.

Next, install kubectl CLI and helm CLI.

Install kubectl with Homebrew.

$ brew install kubernetes-cli

$ brew install kubernetes-cli

Install helm with Homebrew.

$ brew install helm

$ brew install helm

Next, retrieve the web application and additional configuration by cloning the hashicorp/vault-guides repository from GitHub.

$ git clone https://github.com/hashicorp/vault-guides.git

$ git clone https://github.com/hashicorp/vault-guides.git

This repository contains supporting content for all of the Vault learn guides. The content specific to this tutorial can be found in a sub-directory.

Go into the vault-guides/operations/provision-vault/kubernetes/minikube/vault-agent-sidecar directory.

$ cd vault-guides/operations/provision-vault/kubernetes/minikube/vault-agent-sidecar

$ cd vault-guides/operations/provision-vault/kubernetes/minikube/vault-agent-sidecar

Working directory: This tutorial assumes that the remainder of commands are executed in this directory.

»Start Minikube

Minikube is a CLI tool that provisions and manages the lifecycle of single-node Kubernetes clusters locally inside Virtual Machines (VM) on your system.

Start a Kubernetes cluster.

$ minikube start
😄  minikube v1.19.0 on Darwin 11.2.3
✨  Using the docker driver based on existing profile
👍  Starting control plane node minikube in cluster minikube
🔄  Restarting existing docker container for "minikube" ...
🐳  Preparing Kubernetes v1.20.2 on Docker 20.10.5 ...
🔎  Verifying Kubernetes components...
    ▪ Using image gcr.io/k8s-minikube/storage-provisioner:v5
🌟  Enabled addons: storage-provisioner, default-storageclass
🏄  Done! kubectl is now configured to use "minikube" cluster and "default" namespace by default

$ minikube start
😄  minikube v1.19.0 on Darwin 11.2.3
✨  Using the docker driver based on existing profile
👍  Starting control plane node minikube in cluster minikube
🔄  Restarting existing docker container for "minikube" ...
🐳  Preparing Kubernetes v1.20.2 on Docker 20.10.5 ...
🔎  Verifying Kubernetes components...
    ▪ Using image gcr.io/k8s-minikube/storage-provisioner:v5
🌟  Enabled addons: storage-provisioner, default-storageclass
🏄  Done! kubectl is now configured to use "minikube" cluster and "default" namespace by default

The initialization process takes several minutes as it retrieves any necessary dependencies and executes various container images.

Verify the status of the Minikube cluster.

$ minikube status
minikube
type: Control Plane
host: Running
kubelet: Running
apiserver: Running
kubeconfig: Configured

$ minikube status
minikube
type: Control Plane
host: Running
kubelet: Running
apiserver: Running
kubeconfig: Configured

Additional waiting: Even if this last command completed successfully, you may have to wait for Minikube to be available. If an error is displayed, try again after a few minutes.

The host, kubelet, and apiserver report that they are running. The kubectl, a command line interface (CLI) for running commands against Kubernetes cluster, is also configured to communicate with this recently started cluster.

Minikube provides a visual representation of the status in a web-based dashboard. This interface displays the cluster activity in a visual interface that can assist in delving into the issues affecting it.

In another terminal, launch the minikube dashboard.

$ minikube dashboard

$ minikube dashboard

The operating system's default browser opens and displays the dashboard.

Minikube Dashboard

»Install the Vault Helm chart

The recommended way to run Vault on Kubernetes is via the Helm chart. Helm is a package manager that installs and configures all the necessary components to run Vault in several different modes. A Helm chart includes templates that enable conditional and parameterized execution. These parameters can be set through command-line arguments or defined in YAML.

Add the HashiCorp Helm repository.

$ helm repo add hashicorp https://helm.releases.hashicorp.com
"hashicorp" has been added to your repositories

$ helm repo add hashicorp https://helm.releases.hashicorp.com
"hashicorp" has been added to your repositories

Update all the repositories to ensure helm is aware of the latest versions.

$ helm repo update
Hang tight while we grab the latest from your chart repositories...
...Successfully got an update from the "hashicorp" chart repository
Update Complete. ⎈Happy Helming!⎈

$ helm repo update
Hang tight while we grab the latest from your chart repositories...
...Successfully got an update from the "hashicorp" chart repository
Update Complete. ⎈Happy Helming!⎈

Install the latest version of the Vault server running in development mode.

$ helm install vault hashicorp/vault --set "server.dev.enabled=true"
NAME: vault
## ...

$ helm install vault hashicorp/vault --set "server.dev.enabled=true"
NAME: vault
## ...

The Vault pod and Vault Agent Injector pod are deployed in the default namespace.

Display all the pods in the default namespace.

$ kubectl get pods
NAME                                    READY   STATUS    RESTARTS   AGE
vault-0                                 1/1     Running   0          80s
vault-agent-injector-5945fb98b5-tpglz   1/1     Running   0          80s

$ kubectl get pods
NAME                                    READY   STATUS    RESTARTS   AGE
vault-0                                 1/1     Running   0          80s
vault-agent-injector-5945fb98b5-tpglz   1/1     Running   0          80s

The vault-0 pod runs a Vault server in development mode. The vault-agent-injector pod performs the injection based on the annotations present or patched on a deployment.

Development mode: Running a Vault server in development is automatically initialized and unsealed. This is ideal in a learning environment but NOT recommended for a production environment.

Wait until the vault-0 pod and vault-agent-injector pod are running and ready (1/1).

»Set a secret in Vault

The applications that you deploy in the Inject secrets into the pod section expect Vault to store a username and password stored at the path internal/database/config. To create this secret requires that a key-value secret engine is enabled and a username and password is put at the specified path.

Start an interactive shell session on the vault-0 pod.

$ kubectl exec -it vault-0 -- /bin/sh
/ $

$ kubectl exec -it vault-0 -- /bin/sh
/ $

Your system prompt is replaced with a new prompt / $. Commands issued at this prompt are executed on the vault-0 container.

Enable kv-v2 secrets at the path internal.

$ vault secrets enable -path=internal kv-v2
Success! Enabled the kv-v2 secrets engine at: internal/

$ vault secrets enable -path=internal kv-v2
Success! Enabled the kv-v2 secrets engine at: internal/

Learn more: This tutorial focuses on Vault's integration with Kubernetes and not interacting the key-value secrets engine. For more information refer to the Static Secrets: Key/Value Secret tutorial.

Create a secret at path internal/database/config with a username and password.

$ vault kv put internal/database/config username="db-readonly-username" password="db-secret-password"
Key              Value
---              -----
created_time     2020-03-25T19:03:57.127711644Z
deletion_time    n/a
destroyed        false
version          1

$ vault kv put internal/database/config username="db-readonly-username" password="db-secret-password"
Key              Value
---              -----
created_time     2020-03-25T19:03:57.127711644Z
deletion_time    n/a
destroyed        false
version          1

Verify that the secret is defined at the path internal/database/config.

$ vault kv get internal/database/config
====== Metadata ======
Key              Value
---              -----
created_time     2020-03-25T19:03:57.127711644Z
deletion_time    n/a
destroyed        false
version          1

====== Data ======
Key         Value
---         -----
password    db-secret-password
username    db-readonly-username

$ vault kv get internal/database/config
====== Metadata ======
Key              Value
---              -----
created_time     2020-03-25T19:03:57.127711644Z
deletion_time    n/a
destroyed        false
version          1

====== Data ======
Key         Value
---         -----
password    db-secret-password
username    db-readonly-username

The secret is ready for the application.

Lastly, exit the vault-0 pod.

$ exit

$ exit

»Configure Kubernetes authentication

Vault provides a Kubernetes authentication method that enables clients to authenticate with a Kubernetes Service Account Token. This token is provided to each pod when it is created.

Start an interactive shell session on the vault-0 pod.

$ kubectl exec -it vault-0 -- /bin/sh
/ $

$ kubectl exec -it vault-0 -- /bin/sh
/ $

Your system prompt is replaced with a new prompt / $. Commands issued at this prompt are executed on the vault-0 container.

Enable the Kubernetes authentication method.

$ vault auth enable kubernetes
Success! Enabled kubernetes auth method at: kubernetes/

$ vault auth enable kubernetes
Success! Enabled kubernetes auth method at: kubernetes/

Vault accepts a service token from any client in the Kubernetes cluster. During authentication, Vault verifies that the service account token is valid by querying a token review Kubernetes endpoint.

Configure the Kubernetes authentication method to use the location of the Kubernetes API.

For the best compatibility with recent Kubernetes versions, ensure you are using Vault v1.9.3 or greater.

$ vault write auth/kubernetes/config \
    kubernetes_host="https://$KUBERNETES_PORT_443_TCP_ADDR:443"

$ vault write auth/kubernetes/config \
    kubernetes_host="https://$KUBERNETES_PORT_443_TCP_ADDR:443"

Successful output from the command resembles this example:

Success! Data written to: auth/kubernetes/config

Success! Data written to: auth/kubernetes/config

The environment variable KUBERNETES_PORT_443_TCP_ADDR is defined and references the internal network address of the Kubernetes host.

For a client to read the secret data defined at internal/database/config, requires that the read capability be granted for the path internal/data/database/config. This is an example of a policy. A policy defines a set of capabilities.

Write out the policy named internal-app that enables the read capability for secrets at path internal/data/database/config.

$ vault policy write internal-app - <<EOF
path "internal/data/database/config" {
  capabilities = ["read"]
}
EOF

$ vault policy write internal-app - <<EOF
path "internal/data/database/config" {
  capabilities = ["read"]
}
EOF

Create a Kubernetes authentication role named internal-app.

$ vault write auth/kubernetes/role/internal-app \
    bound_service_account_names=internal-app \
    bound_service_account_namespaces=default \
    policies=internal-app \
    ttl=24h

$ vault write auth/kubernetes/role/internal-app \
    bound_service_account_names=internal-app \
    bound_service_account_namespaces=default \
    policies=internal-app \
    ttl=24h

Successful output from the command resembles this example:

Success! Data written to: auth/kubernetes/role/internal-app

Success! Data written to: auth/kubernetes/role/internal-app

The role connects the Kubernetes service account, internal-app, and namespace, default, with the Vault policy, internal-app. The tokens returned after authentication are valid for 24 hours.

Lastly, exit the vault-0 pod.

$ exit

$ exit

»Define a Kubernetes service account

The Vault Kubernetes authentication role defined a Kubernetes service account named internal-app.

Get all the service accounts in the default namespace.

$ kubectl get serviceaccounts
NAME                   SECRETS   AGE
default                1         43m
vault                  1         34m
vault-agent-injector   1         34m

$ kubectl get serviceaccounts
NAME                   SECRETS   AGE
default                1         43m
vault                  1         34m
vault-agent-injector   1         34m

Create a Kubernetes service account named internal-app in the default namespace.

$ kubectl create sa internal-app

$ kubectl create sa internal-app

Verify that the service account has been created.

$ kubectl get serviceaccounts
NAME                   SECRETS   AGE
default                1         52m
internal-app           1         13s
vault                  1         43m
vault-agent-injector   1         43m

$ kubectl get serviceaccounts
NAME                   SECRETS   AGE
default                1         52m
internal-app           1         13s
vault                  1         43m
vault-agent-injector   1         43m

The name of the service account here aligns with the name assigned to the bound_service_account_names field when the internal-app role was created.

»Launch an application

We've created a sample application, published it to DockerHub, and created a Kubernetes deployment that launches this application.

Display the deployment for the orgchart application.

$ cat deployment-orgchart.yaml

$ cat deployment-orgchart.yaml

apiVersion: apps/v1
kind: Deployment
metadata:
  name: orgchart
  labels:
    app: orgchart
spec:
  selector:
    matchLabels:
      app: orgchart
  replicas: 1
  template:
    metadata:
      annotations:
      labels:
        app: orgchart
    spec:
      serviceAccountName: internal-app
      containers:
        - name: orgchart
          image: jweissig/app:0.0.1
deployment-orgchart.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: orgchart
  labels:
    app: orgchart
spec:
  selector:
    matchLabels:
      app: orgchart
  replicas: 1
  template:
    metadata:
      annotations:
      labels:
        app: orgchart
    spec:
      serviceAccountName: internal-app
      containers:
        - name: orgchart
          image: jweissig/app:0.0.1

The name of this deployment is orgchart. The spec.template.spec.serviceAccountName defines the service account internal-app to run this container.

Apply the deployment defined in deployment-orgchart.yaml.

$ kubectl apply --filename deployment-orgchart.yaml
deployment.apps/orgchart created

$ kubectl apply --filename deployment-orgchart.yaml
deployment.apps/orgchart created

Get all the pods in the default namespace.

$ kubectl get pods
NAME                                    READY   STATUS    RESTARTS   AGE
orgchart-69697d9598-l878s               1/1     Running   0          18s
vault-0                                 1/1     Running   0          58m
vault-agent-injector-5945fb98b5-tpglz   1/1     Running   0          58m

$ kubectl get pods
NAME                                    READY   STATUS    RESTARTS   AGE
orgchart-69697d9598-l878s               1/1     Running   0          18s
vault-0                                 1/1     Running   0          58m
vault-agent-injector-5945fb98b5-tpglz   1/1     Running   0          58m

The orgchart pod is displayed here as the pod prefixed with orgchart.

Additional waiting: The deployment of the pod requires the retrieval of the application container from Docker Hub. This displays the STATUS of ContainerCreating. The pod reports that it is not ready (0/1).

The Vault-Agent injector looks for deployments that define specific annotations. None of these annotations exist in the current deployment. This means that no secrets are present on the orgchart container in the orgchart pod.

Verify that no secrets are written to the orgchart container in the orgchart pod.

$ kubectl exec \
    $(kubectl get pod -l app=orgchart -o jsonpath="{.items[0].metadata.name}") \
    --container orgchart -- ls /vault/secrets

$ kubectl exec \
    $(kubectl get pod -l app=orgchart -o jsonpath="{.items[0].metadata.name}") \
    --container orgchart -- ls /vault/secrets

The output displays that there is no such file or directory named /vault/secrets:

ls: /vault/secrets: No such file or directory
command terminated with exit code 1

ls: /vault/secrets: No such file or directory
command terminated with exit code 1

»Inject secrets into the pod

The deployment is running the pod with the internal-app Kubernetes service account in the default namespace. The Vault Agent Injector only modifies a deployment if it contains a specific set of annotations. An existing deployment may have its definition patched to include the necessary annotations.

Display the deployment patch patch-inject-secrets.yaml.

$ cat patch-inject-secrets.yaml

$ cat patch-inject-secrets.yaml

spec:
  template:
    metadata:
      annotations:
        vault.hashicorp.com/agent-inject: 'true'
        vault.hashicorp.com/role: 'internal-app'
        vault.hashicorp.com/agent-inject-secret-database-config.txt: 'internal/data/database/config'
patch-inject-secrets.yaml
spec:
  template:
    metadata:
      annotations:
        vault.hashicorp.com/agent-inject: 'true'
        vault.hashicorp.com/role: 'internal-app'
        vault.hashicorp.com/agent-inject-secret-database-config.txt: 'internal/data/database/config'

These annotations define a partial structure of the deployment schema and are prefixed with vault.hashicorp.com.

  • agent-inject enables the Vault Agent Injector service
  • role is the Vault Kubernetes authentication role
  • agent-inject-secret-FILEPATH prefixes the path of the file, database-config.txt written to the /vault/secrets directory. The value is the path to the secret defined in Vault.

Patch the orgchart deployment defined in patch-inject-secrets.yaml.

$ kubectl patch deployment orgchart --patch "$(cat patch-inject-secrets.yaml)"
deployment.apps/orgchart patched

$ kubectl patch deployment orgchart --patch "$(cat patch-inject-secrets.yaml)"
deployment.apps/orgchart patched

A new orgchart pod starts alongside the existing pod. When it is ready the original terminates and removes itself from the list of active pods.

Get all the pods in the default namespace.

$ kubectl get pods
NAME                                    READY   STATUS     RESTARTS   AGE
orgchart-599cb74d9c-s8hhm               0/2     Init:0/1   0          23s
orgchart-69697d9598-l878s               1/1     Running    0          20m
vault-0                                 1/1     Running    0          78m
vault-agent-injector-5945fb98b5-tpglz   1/1     Running    0          78m

$ kubectl get pods
NAME                                    READY   STATUS     RESTARTS   AGE
orgchart-599cb74d9c-s8hhm               0/2     Init:0/1   0          23s
orgchart-69697d9598-l878s               1/1     Running    0          20m
vault-0                                 1/1     Running    0          78m
vault-agent-injector-5945fb98b5-tpglz   1/1     Running    0          78m

Wait until the re-deployed orgchart pod reports that it is Running and ready (2/2).

This new pod now launches two containers. The application container, named orgchart, and the Vault Agent container, named vault-agent.

Display the logs of the vault-agent container in the new orgchart pod.

$ kubectl logs \
    $(kubectl get pod -l app=orgchart -o jsonpath="{.items[0].metadata.name}") \
    --container vault-agent

$ kubectl logs \
    $(kubectl get pod -l app=orgchart -o jsonpath="{.items[0].metadata.name}") \
    --container vault-agent

Vault Agent manages the token lifecycle and the secret retrieval. The secret is rendered in the orgchart container at the path /vault/secrets/database-config.txt.

Finally, display the secret written to the orgchart container.

$ kubectl exec \
    $(kubectl get pod -l app=orgchart -o jsonpath="{.items[0].metadata.name}") \
    --container orgchart -- cat /vault/secrets/database-config.txt

$ kubectl exec \
    $(kubectl get pod -l app=orgchart -o jsonpath="{.items[0].metadata.name}") \
    --container orgchart -- cat /vault/secrets/database-config.txt

The unformatted secret data is present on the container:

data: map[password:db-secret-password username:db-readonly-user]
metadata: map[created_time:2019-12-20T18:17:50.930264759Z deletion_time: destroyed:false version:2]

data: map[password:db-secret-password username:db-readonly-user]
metadata: map[created_time:2019-12-20T18:17:50.930264759Z deletion_time: destroyed:false version:2]

»Apply a template to the injected secrets

The structure of the injected secrets may need to be structured in a way for an application to use. Before writing the secrets to the file system a template can structure the data. To apply this template a new set of annotations need to be applied.

Display the annotations file that contains a template definition.

$ cat patch-inject-secrets-as-template.yaml

$ cat patch-inject-secrets-as-template.yaml

spec:
  template:
    metadata:
      annotations:
        vault.hashicorp.com/agent-inject: 'true'
        vault.hashicorp.com/agent-inject-status: 'update'
        vault.hashicorp.com/role: 'internal-app'
        vault.hashicorp.com/agent-inject-secret-database-config.txt: 'internal/data/database/config'
        vault.hashicorp.com/agent-inject-template-database-config.txt: |
          {{- with secret "internal/data/database/config" -}}
          postgresql://{{ .Data.data.username }}:{{ .Data.data.password }}@postgres:5432/wizard
          {{- end -}}
patch-inject-secrets-as-template.yaml
spec:
  template:
    metadata:
      annotations:
        vault.hashicorp.com/agent-inject: 'true'
        vault.hashicorp.com/agent-inject-status: 'update'
        vault.hashicorp.com/role: 'internal-app'
        vault.hashicorp.com/agent-inject-secret-database-config.txt: 'internal/data/database/config'
        vault.hashicorp.com/agent-inject-template-database-config.txt: |
          {{- with secret "internal/data/database/config" -}}
          postgresql://{{ .Data.data.username }}:{{ .Data.data.password }}@postgres:5432/wizard
          {{- end -}}

This patch contains two new annotations:

The template formats the username and password as a PostgreSQL connection string.

Apply the updated annotations.

$ kubectl patch deployment orgchart --patch "$(cat patch-inject-secrets-as-template.yaml)"
deployment.apps/exampleapp patched

$ kubectl patch deployment orgchart --patch "$(cat patch-inject-secrets-as-template.yaml)"
deployment.apps/exampleapp patched

Get all the pods in the default namespace.

$ kubectl get pods
NAME                                    READY   STATUS    RESTARTS   AGE
orgchart-554db4579d-w6565               2/2     Running   0          16s
vault-0                                 1/1     Running   0          126m
vault-agent-injector-5945fb98b5-tpglz   1/1     Running   0          126m

$ kubectl get pods
NAME                                    READY   STATUS    RESTARTS   AGE
orgchart-554db4579d-w6565               2/2     Running   0          16s
vault-0                                 1/1     Running   0          126m
vault-agent-injector-5945fb98b5-tpglz   1/1     Running   0          126m

Wait until the re-deployed orgchart pod reports that it is Running and ready (2/2).

Finally, display the secret written to the orgchart container in the orgchart pod.

$ kubectl exec \
    $(kubectl get pod -l app=orgchart -o jsonpath="{.items[0].metadata.name}") \
    -c orgchart -- cat /vault/secrets/database-config.txt

$ kubectl exec \
    $(kubectl get pod -l app=orgchart -o jsonpath="{.items[0].metadata.name}") \
    -c orgchart -- cat /vault/secrets/database-config.txt

The secrets are rendered in a PostgreSQL connection string is present on the container:

postgresql://db-readonly-user:db-secret-password@postgres:5432/wizard

postgresql://db-readonly-user:db-secret-password@postgres:5432/wizard

»Pod with annotations

The annotations may patch these secrets into any deployment. Pods require that the annotations be included in their initial definition.

Display the pod definition for the payroll application.

$ cat pod-payroll.yaml

$ cat pod-payroll.yaml

apiVersion: v1
kind: Pod
metadata:
  name: payroll
  labels:
    app: payroll
  annotations:
    vault.hashicorp.com/agent-inject: 'true'
    vault.hashicorp.com/role: 'internal-app'
    vault.hashicorp.com/agent-inject-secret-database-config.txt: 'internal/data/database/config'
    vault.hashicorp.com/agent-inject-template-database-config.txt: |
      {{- with secret "internal/data/database/config" -}}
      postgresql://{{ .Data.data.username }}:{{ .Data.data.password }}@postgres:5432/wizard
      {{- end -}}
spec:
  serviceAccountName: internal-app
  containers:
    - name: payroll
      image: jweissig/app:0.0.1
pod-payroll.yaml
apiVersion: v1
kind: Pod
metadata:
  name: payroll
  labels:
    app: payroll
  annotations:
    vault.hashicorp.com/agent-inject: 'true'
    vault.hashicorp.com/role: 'internal-app'
    vault.hashicorp.com/agent-inject-secret-database-config.txt: 'internal/data/database/config'
    vault.hashicorp.com/agent-inject-template-database-config.txt: |
      {{- with secret "internal/data/database/config" -}}
      postgresql://{{ .Data.data.username }}:{{ .Data.data.password }}@postgres:5432/wizard
      {{- end -}}
spec:
  serviceAccountName: internal-app
  containers:
    - name: payroll
      image: jweissig/app:0.0.1

Apply the pod defined in pod-payroll.yaml.

$ kubectl apply --filename pod-payroll.yaml
pod/payroll created

$ kubectl apply --filename pod-payroll.yaml
pod/payroll created

Get all the pods in the default namespace.

$ kubectl get pods
NAME                                    READY   STATUS    RESTARTS   AGE
orgchart-554db4579d-w6565               2/2     Running   0          29m
payroll                                 2/2     Running   0          12s
vault-0                                 1/1     Running   0          155m
vault-agent-injector-5945fb98b5-tpglz   1/1     Running   0          155m

$ kubectl get pods
NAME                                    READY   STATUS    RESTARTS   AGE
orgchart-554db4579d-w6565               2/2     Running   0          29m
payroll                                 2/2     Running   0          12s
vault-0                                 1/1     Running   0          155m
vault-agent-injector-5945fb98b5-tpglz   1/1     Running   0          155m

Wait until the payroll pod reports that it is Running and ready (2/2).

Finally, display the secret written to the payroll container in the payroll pod.

$ kubectl exec \
    payroll \
    --container payroll -- cat /vault/secrets/database-config.txt

$ kubectl exec \
    payroll \
    --container payroll -- cat /vault/secrets/database-config.txt

The secrets are rendered in a PostgreSQL connection string is present on the container:

postgresql://db-readonly-user:db-secret-password@postgres:5432/wizard

postgresql://db-readonly-user:db-secret-password@postgres:5432/wizard

»Secrets are bound to the service account

Pods run with a Kubernetes service account other than the ones defined in the Vault Kubernetes authentication role are NOT able to access the secrets defined at that path.

Display the deployment and service account for the website application.

$ cat deployment-website.yaml

$ cat deployment-website.yaml

apiVersion: apps/v1
kind: Deployment
metadata:
  name: website
  labels:
    app: website
spec:
  selector:
    matchLabels:
      app: website
  replicas: 1
  template:
    metadata:
      annotations:
        vault.hashicorp.com/agent-inject: 'true'
        vault.hashicorp.com/role: 'internal-app'
        vault.hashicorp.com/agent-inject-secret-database-config.txt: 'internal/data/database/config'
        vault.hashicorp.com/agent-inject-template-database-config.txt: |
          {{- with secret "internal/data/database/config" -}}
          postgresql://{{ .Data.data.username }}:{{ .Data.data.password }}@postgres:5432/wizard
          {{- end -}}
      labels:
        app: website
    spec:
      # This service account does not have permission to request the secrets.
      serviceAccountName: website
      containers:
        - name: website
          image: jweissig/app:0.0.1
---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: website
deployment-website.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: website
  labels:
    app: website
spec:
  selector:
    matchLabels:
      app: website
  replicas: 1
  template:
    metadata:
      annotations:
        vault.hashicorp.com/agent-inject: 'true'
        vault.hashicorp.com/role: 'internal-app'
        vault.hashicorp.com/agent-inject-secret-database-config.txt: 'internal/data/database/config'
        vault.hashicorp.com/agent-inject-template-database-config.txt: |
          {{- with secret "internal/data/database/config" -}}
          postgresql://{{ .Data.data.username }}:{{ .Data.data.password }}@postgres:5432/wizard
          {{- end -}}
      labels:
        app: website
    spec:
      # This service account does not have permission to request the secrets.
      serviceAccountName: website
      containers:
        - name: website
          image: jweissig/app:0.0.1
---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: website

Apply the deployment and service account defined in deployment-website.yaml.

$ kubectl apply --filename deployment-website.yaml
deployment.apps/website created
serviceaccount/website created

$ kubectl apply --filename deployment-website.yaml
deployment.apps/website created
serviceaccount/website created

Get all the pods in the default namespace.

$ kubectl get pods
NAME                                    READY   STATUS     RESTARTS   AGE
orgchart-554db4579d-w6565               2/2     Running    0          29m
payroll                                 2/2     Running    0          12s
vault-0                                 1/1     Running    0          155m
vault-agent-injector-5945fb98b5-tpglz   1/1     Running    0          155m
website-7fc8b69645-527rf                0/2     Init:0/1   0          76s

$ kubectl get pods
NAME                                    READY   STATUS     RESTARTS   AGE
orgchart-554db4579d-w6565               2/2     Running    0          29m
payroll                                 2/2     Running    0          12s
vault-0                                 1/1     Running    0          155m
vault-agent-injector-5945fb98b5-tpglz   1/1     Running    0          155m
website-7fc8b69645-527rf                0/2     Init:0/1   0          76s

The website deployment creates a pod but it is NEVER ready.

Display the logs of the vault-agent-init container in the website pod.

$ kubectl logs \
    $(kubectl get pod -l app=website -o jsonpath="{.items[0].metadata.name}") \
    --container vault-agent-init

$ kubectl logs \
    $(kubectl get pod -l app=website -o jsonpath="{.items[0].metadata.name}") \
    --container vault-agent-init

The initialization process failed because the service account name is not authorized:

...
[INFO]  auth.handler: authenticating
[ERROR] auth.handler: error authenticating: error="Error making API request.

URL: PUT http://vault.default.svc:8200/v1/auth/kubernetes/login
Code: 500. Errors:

* service account name not authorized" backoff=1.562132589

...
[INFO]  auth.handler: authenticating
[ERROR] auth.handler: error authenticating: error="Error making API request.

URL: PUT http://vault.default.svc:8200/v1/auth/kubernetes/login
Code: 500. Errors:

* service account name not authorized" backoff=1.562132589

The service account, external-app is not assigned to any Vault Kubernetes authentication role. This failure to authenticate causes the deployment to fail initialization.

Display the deployment patch patch-website.yaml.

$ cat patch-website.yaml

$ cat patch-website.yaml

spec:
  template:
    spec:
      serviceAccountName: internal-app
patch-website.yaml
spec:
  template:
    spec:
      serviceAccountName: internal-app

The patch modifies the deployment definition to use the service account internal-app. This Kubernetes service account is authorized by the Vault Kubernetes authentication role.

Patch the website deployment defined in patch-website.yaml.

$ kubectl patch deployment website --patch "$(cat patch-website.yaml)"

$ kubectl patch deployment website --patch "$(cat patch-website.yaml)"

Get all the pods in the default namespace.

$ kubectl get pods
NAME                                    READY   STATUS     RESTARTS   AGE
orgchart-554db4579d-w6565               2/2     Running    0          29m
payroll                                 2/2     Running    0          12s
vault-0                                 1/1     Running    0          155m
vault-agent-injector-5945fb98b5-tpglz   1/1     Running    0          155m
website-788d689b87-tll2r                2/2     Running    0          27s

$ kubectl get pods
NAME                                    READY   STATUS     RESTARTS   AGE
orgchart-554db4579d-w6565               2/2     Running    0          29m
payroll                                 2/2     Running    0          12s
vault-0                                 1/1     Running    0          155m
vault-agent-injector-5945fb98b5-tpglz   1/1     Running    0          155m
website-788d689b87-tll2r                2/2     Running    0          27s

Wait until the website pod reports that it is Running and ready (2/2).

Finally, display the secret written to the website container in the website pod.

$ kubectl exec \
    $(kubectl get pod -l app=website -o jsonpath="{.items[0].metadata.name}") \
    --container website -- cat /vault/secrets/database-config.txt

$ kubectl exec \
    $(kubectl get pod -l app=website -o jsonpath="{.items[0].metadata.name}") \
    --container website -- cat /vault/secrets/database-config.txt

The secrets are rendered in a PostgreSQL connection string is present on the container:

postgresql://db-readonly-user:db-secret-password@postgres:5432/wizard

postgresql://db-readonly-user:db-secret-password@postgres:5432/wizard

Vault Kubernetes Roles: Alternatively, you can define a new Vault Kubernetes role, that enables the original service account access, and patch the deployment.

»Secrets are bound to the namespace

Pods run in a namespace other than the ones defined in the Vault Kubernetes authentication role are NOT able to access the secrets defined at that path.

Create the offsite namespace.

$ kubectl create namespace offsite
namespace/offsite created

$ kubectl create namespace offsite
namespace/offsite created

Set the current context to the offsite namespace.

$ kubectl config set-context --current --namespace offsite
Context "minikube" modified.

$ kubectl config set-context --current --namespace offsite
Context "minikube" modified.

Create a Kubernetes service account named internal-app in the offsite namespace.

$ kubectl create sa internal-app
serviceaccount/internal-app created

$ kubectl create sa internal-app
serviceaccount/internal-app created

Display the deployment for the issues application.

$ cat deployment-issues.yaml

$ cat deployment-issues.yaml

apiVersion: apps/v1
kind: Deployment
metadata:
  name: issues
  labels:
    app: issues
spec:
  selector:
    matchLabels:
      app: issues
  replicas: 1
  template:
    metadata:
      annotations:
        vault.hashicorp.com/agent-inject: 'true'
        vault.hashicorp.com/role: 'internal-app'
        vault.hashicorp.com/agent-inject-secret-database-config.txt: 'internal/data/database/config'
        vault.hashicorp.com/agent-inject-template-database-config.txt: |
          {{- with secret "internal/data/database/config" -}}
          postgresql://{{ .Data.data.username }}:{{ .Data.data.password }}@postgres:5432/wizard
          {{- end -}}
      labels:
        app: issues
    spec:
      serviceAccountName: internal-app
      containers:
        - name: issues
          image: jweissig/app:0.0.1
deployment-issues.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: issues
  labels:
    app: issues
spec:
  selector:
    matchLabels:
      app: issues
  replicas: 1
  template:
    metadata:
      annotations:
        vault.hashicorp.com/agent-inject: 'true'
        vault.hashicorp.com/role: 'internal-app'
        vault.hashicorp.com/agent-inject-secret-database-config.txt: 'internal/data/database/config'
        vault.hashicorp.com/agent-inject-template-database-config.txt: |
          {{- with secret "internal/data/database/config" -}}
          postgresql://{{ .Data.data.username }}:{{ .Data.data.password }}@postgres:5432/wizard
          {{- end -}}
      labels:
        app: issues
    spec:
      serviceAccountName: internal-app
      containers:
        - name: issues
          image: jweissig/app:0.0.1

Apply the deployment defined in deployment-issues.yaml.

$ kubectl apply --filename deployment-issues.yaml
deployment.apps/issues created

$ kubectl apply --filename deployment-issues.yaml
deployment.apps/issues created

Get all the pods in the offsite namespace.

$ kubectl get pods
NAME                      READY   STATUS     RESTARTS   AGE
issues-79d8bf7cdf-dkdlq   0/2     Init:0/1   0          3s

$ kubectl get pods
NAME                      READY   STATUS     RESTARTS   AGE
issues-79d8bf7cdf-dkdlq   0/2     Init:0/1   0          3s

Current context: The same command is issued but the results are different because you are now in a different namespace.

The issues deployment creates a pod but it is NEVER ready.

Display the logs of the vault-agent-init container in the issues pod.

$ kubectl logs \
    $(kubectl get pod -l app=issues -o jsonpath="{.items[0].metadata.name}") \
    --container vault-agent-init

$ kubectl logs \
    $(kubectl get pod -l app=issues -o jsonpath="{.items[0].metadata.name}") \
    --container vault-agent-init

The initialization process fails because the namespace is not authorized:

...
[INFO]  auth.handler: authenticating
[ERROR] auth.handler: error authenticating: error="Error making API request.

URL: PUT http://vault.default.svc:8200/v1/auth/kubernetes/login
Code: 500. Errors:

* namespace not authorized" backoff=1.9882590740000001

...
[INFO]  auth.handler: authenticating
[ERROR] auth.handler: error authenticating: error="Error making API request.

URL: PUT http://vault.default.svc:8200/v1/auth/kubernetes/login
Code: 500. Errors:

* namespace not authorized" backoff=1.9882590740000001

The namespace, offsite is not assigned to any Vault Kubernetes authentication role. This failure to authenticate causes the deployment to fail initialization.

Start an interactive shell session on the vault-0 pod in the default namespace.

$ kubectl exec --namespace default -it vault-0 -- /bin/sh
/ $

$ kubectl exec --namespace default -it vault-0 -- /bin/sh
/ $

Your system prompt is replaced with a new prompt / $. Commands issued at this prompt are executed on the vault-0 container.

Create a Kubernetes authentication role named offsite-app.

$ vault write auth/kubernetes/role/offsite-app \
    bound_service_account_names=internal-app \
    bound_service_account_namespaces=offsite \
    policies=internal-app \
    ttl=24h

$ vault write auth/kubernetes/role/offsite-app \
    bound_service_account_names=internal-app \
    bound_service_account_namespaces=offsite \
    policies=internal-app \
    ttl=24h

Successful output from the command resembles this example:

Success! Data written to: auth/kubernetes/role/offsite-app

Success! Data written to: auth/kubernetes/role/offsite-app

Exit the vault-0 pod.

$ exit

$ exit

Display the deployment patch patch-issues.yaml.

$ cat patch-issues.yaml

$ cat patch-issues.yaml

spec:
  template:
    metadata:
      annotations:
        vault.hashicorp.com/agent-inject: 'true'
        vault.hashicorp.com/agent-inject-status: 'update'
        vault.hashicorp.com/role: 'offsite-app'
        vault.hashicorp.com/agent-inject-secret-database-config.txt: 'internal/data/database/config'
        vault.hashicorp.com/agent-inject-template-database-config.txt: |
          {{- with secret "internal/data/database/config" -}}
          postgresql://{{ .Data.data.username }}:{{ .Data.data.password }}@postgres:5432/wizard
          {{- end -}}
patch-issues.yaml
spec:
  template:
    metadata:
      annotations:
        vault.hashicorp.com/agent-inject: 'true'
        vault.hashicorp.com/agent-inject-status: 'update'
        vault.hashicorp.com/role: 'offsite-app'
        vault.hashicorp.com/agent-inject-secret-database-config.txt: 'internal/data/database/config'
        vault.hashicorp.com/agent-inject-template-database-config.txt: |
          {{- with secret "internal/data/database/config" -}}
          postgresql://{{ .Data.data.username }}:{{ .Data.data.password }}@postgres:5432/wizard
          {{- end -}}

The patch performs an update to set the vault.hashicorp.com/role to the Vault Kubernetes role offsite-app.

Patch the issues deployment defined in patch-issues.yaml.

$ kubectl patch deployment issues --patch "$(cat patch-issues.yaml)"
deployment.apps/issues patched

$ kubectl patch deployment issues --patch "$(cat patch-issues.yaml)"
deployment.apps/issues patched

A new issues pod starts alongside the existing pod. When it is ready the original terminates and removes itself from the list of active pods.

Get all the pods in the offsite namespace.

$ kubectl get pods
NAME                      READY   STATUS    RESTARTS   AGE
issues-7fd66f98f6-ffzh7   2/2     Running   0          94s

$ kubectl get pods
NAME                      READY   STATUS    RESTARTS   AGE
issues-7fd66f98f6-ffzh7   2/2     Running   0          94s

Wait until the re-deployed issues pod reports that it is Running and ready (2/2).

Finally, display the secret written to the issues container in the issues pod.

$ kubectl exec \
    $(kubectl get pod -l app=issues -o jsonpath="{.items[0].metadata.name}") \
    --container issues -- cat /vault/secrets/database-config.txt

$ kubectl exec \
    $(kubectl get pod -l app=issues -o jsonpath="{.items[0].metadata.name}") \
    --container issues -- cat /vault/secrets/database-config.txt

The secrets are rendered in a PostgreSQL connection string is present on the container:

postgresql://db-readonly-user:db-secret-password@postgres:5432/wizard

postgresql://db-readonly-user:db-secret-password@postgres:5432/wizard

»Next Steps

You launched Vault and the injector service with the Vault Helm chart. Learn more about the Vault Helm chart by reading the documentation, exploring the project source code, reading the blog post announcing the "Injecting Vault Secrets into Kubernetes Pods via a Sidecar", or the documentation for Agent Sidecar Injector

Then you deployed several applications to demonstrate how this new injector service retrieves and writes these secrets for the applications to use. Explore how pods can retrieve them directly via network requests or secrets mounted on ephemeral volumes.

stdin: is not a tty