Virtual Event
Join us for the next HashiConf Digital October 12-15, 2020 Register for Free

Kubernetes

Mount Vault Secrets through Container Storage Interface (CSI) Volume

Kubernetes application pods that rely on Vault to manage their secrets can retrieve them directly via network requests or maintained on a mounted file system through the Vault Injector service via annotations or attached as ephemeral volumes. This approach of employing empheral volumes to store secrets is a feature of the Secrets Store extension to the Kubernetes Container Storage Interface (CSI) driver.

In this guide, you will setup Vault and its dependencies with a Helm chart. Then enable and configure the secrets store CSI driver to create a volume that contains a secret that you will mount to an application pod.

»Prerequisites

This guide requires the Kubernetes command-line interface (CLI) and the Helm CLI installed, Minikube, and additional configuration to bring it all together.

Online tutorial: An interactive tutorial is also available if you do not wish to install the following resources. Click the Show Tutorial button to launch the tutorial.

This guide was last tested 20 Jun 2020 on macOS 10.15.5 using this configuration.

Docker version.

$ docker version
Client: Docker Engine - Community
  Version:          19.03.8
  # ...

Minikube version.

$ minikube version
minikube version: v1.11.0
commit: 57e2f55f47effe9ce396cea42a1e0eb4f611ebbd

Helm version.

$ helm version
version.BuildInfo{Version:"v3.2.1", GitCommit:"fe51cd1e31e6a202cba7dead9552a6d418ded79a", GitTreeState:"clean", GoVersion:"go1.13.10"}

These are recommended software versions and the output displayed may vary depending on your environment and the software versions you use.

First, follow the directions to install Minikube, including VirtualBox or similar.

Next, install kubectl CLI and helm CLI.

Install kubectl with Homebrew.

$ brew install kubernetes-cli

Install helm with Homebrew.

$ brew install helm

»Start Minikube

Minikube is a CLI tool that provisions and manages the lifecycle of single-node Kubernetes clusters. These clusters are run locally inside Virtual Machines (VM).

Start a Kubernetes cluster.

$ minikube start
😄  minikube v1.11.0 on Darwin 10.15.5
✨  Automatically selected the hyperkit driver
👍  Starting control plane node minikube in cluster minikube
🔥  Creating hyperkit VM (CPUs=2, Memory=4000MB, Disk=20000MB) ...
🐳  Preparing Kubernetes v1.18.3 on Docker 19.03.8 ...
🔎  Verifying Kubernetes components...
🌟  Enabled addons: default-storageclass, storage-provisioner
🏄  Done! kubectl is now configured to use "minikube"

Verify the status of the Minikube cluster.

$ minikube status
minikube
type: Control Plane
host: Running
kubelet: Running
apiserver: Running
kubeconfig: Configured

The host, kubelet, apiserver report that they are running. The kubectl, a command line interface (CLI) for running commands against Kubernetes cluster, is also configured to communicate with this recently started cluster.

»Install the Vault Helm chart

Vault manages the secrets that are written to these mountable volumes. To provide these secrets a single Vault server is required. For this demonstration Vault can be run in development mode to automatically handle initialization, unsealing, and setup of a KV secrets engine.

Add the HashiCorp Helm repository.

$ helm repo add hashicorp https://helm.releases.hashicorp.com
"hashicorp" has been added to your repositories

Install the latest version of the Vault Helm chart running in development mode with the injector service disabled.

$ helm install vault hashicorp/vault \
    --set "server.dev.enabled=true" \
    --set "injector.enabled=false"
NAME: vault
## ...

The Vault server runs in development mode on a single pod server.dev.enabled=true. By default the Helm chart starts a Vault Agent Injector pod but that is disabled injector.enabled=false.

Display all the pods within the default namespace.

$ kubectl get pods
NAME      READY   STATUS    RESTARTS   AGE
vault-0   1/1     Running   0          87s

Wait until the vault-0 pod is running and ready (1/1).

»Set a secret in Vault

The volume mounted to the pod in the Create a pod with secret mounted section expects a secret stored at the path secret/data/db-pass. When Vault is run in development a KV secret engine is enabled at the path /secret.

First, start an interactive shell session on the vault-0 pod.

$ kubectl exec -it vault-0 -- /bin/sh
/ $

Your system prompt is replaced with a new prompt / $. Commands issued at this prompt are executed on the vault-0 container.

Create a secret at the path secret/db-pass with a password.

$ vault kv put secret/db-pass password="db-secret-password"
Key              Value
---              -----
created_time     2020-05-30T16:58:54.295890646Z
deletion_time    n/a
destroyed        false
version          1

Verify that the secret is readable at the path secret/db-pass.

$ vault kv get secret/db-pass
====== Metadata ======
Key              Value
---              -----
created_time     2020-05-30T16:58:54.295890646Z
deletion_time    n/a
destroyed        false
version          1

====== Data ======
Key         Value
---         -----
password    db-secret-password

Lastly, exit the vault-0 pod.

$ exit

»Configure Kubernetes authentication

Vault provides a Kubernetes authentication method that enables clients to authenticate with a Kubernetes Service Account Token. The Kubernetes resources that access the secret and create the volume authenticate through this method through a role.

First, start an interactive shell session on the vault-0 pod.

$ kubectl exec -it vault-0 -- /bin/sh
/ $

Your system prompt is replaced with a new prompt / $. Commands issued at this prompt are executed on the vault-0 container.

Enable the Kubernetes authentication method.

$ vault auth enable kubernetes
Success! Enabled kubernetes auth method at: kubernetes/

Configure the Kubernetes authentication method to use the service account token, the location of the Kubernetes host, and its certificate.

$ vault write auth/kubernetes/config \
    token_reviewer_jwt="$(cat /var/run/secrets/kubernetes.io/serviceaccount/token)" \
    kubernetes_host="https://$KUBERNETES_PORT_443_TCP_ADDR:443" \
    kubernetes_ca_cert=@/var/run/secrets/kubernetes.io/serviceaccount/ca.crt
Success! Data written to: auth/kubernetes/config

The token_reviewer_jwt and kubernetes_ca_cert files written to the container by Kubernetes. The environment variable KUBERNETES_PORT_443_TCP_ADDR references the internal network address of the Kubernetes host.

For the Kubernetes-Secrets-Store-CSI-Driver to read the secrets requires that it has read permissions of all mounts and access to the secret itself.

Write out the policy named internal-app.

$ vault policy write internal-app - <<EOF
path "sys/mounts" {
  capabilities = ["read"]
}

path "secret/data/db-pass" {
  capabilities = ["read"]
}
EOF
Success! Uploaded policy: internal-app

Currently the Vault extension of the Kubernetes-Secrets-Store-CSI-Driver only supports the KV Secrets Engine. This extension verifies that the requested secret belongs to a supported engine by reading the mounted secrets engines. The data of kv-v2 requires that an additional path element of data is included after its mount path (in this case, secret/).

Finally, create a Kubernetes authentication role named database that binds this policy with a Kubernetes service account named secrets-store-csi-driver.

$ vault write auth/kubernetes/role/database \
    bound_service_account_names=secrets-store-csi-driver \
    bound_service_account_namespaces=default \
    policies=internal-app \
    ttl=20m
Success! Data written to: auth/kubernetes/role/database

The role connects the Kubernetes service account, secrets-store-csi-driver, in the namespace, default, with the Vault policy, internal-app. The tokens returned after authentication are valid for 20 minutes. This Kubernetes service account name, secrets-store-csi-driver, is created in the Install the secrets store CSI driver section when the secrets store CSI driver Helm chart is installed.

Lastly, exit the vault-0 pod.

$ exit

»Install the secrets store CSI driver

The Secrets Store CSI driver secrets-store.csi.k8s.io allows Kubernetes to mount multiple secrets, keys, and certs stored in enterprise-grade external secrets stores into their pods as a volume. Once the Volume is attached, the data in it is mounted into the container's file system.

Add the Secrets Store CSI driver Helm repository.

$ helm repo add secrets-store-csi-driver https://raw.githubusercontent.com/kubernetes-sigs/secrets-store-csi-driver/master/charts
"secrets-store-csi-driver" has been added to your repositories

Install the latest version of the Kubernetes-Secrets-Store-CSI-Driver.

$ helm install csi secrets-store-csi-driver/secrets-store-csi-driver
NAME: csi
## ...

The csi-secrets-store-csi-driver pod is deployed in the default namespace.

Get all the pods within the default namespace.

$ kubectl get pods
NAME                                 READY   STATUS    RESTARTS   AGE
csi-secrets-store-csi-driver-6rf2k   3/3     Running   0          29s
vault-0                              1/1     Running   0          23m

Wait until the csi-secrets-store-csi-driver pod is running and ready (3/3).

»Install the provider-vault executable

The Secrets Store CSI driver enables extension through providers. A provider is launched as a Kubernetes DaemonSet alongside of Secrets Store CSI driver DaemonSet.

Create the DaemonSet that installs the provider-vault executable for the Kubernetes-Secrets-Store-CSI-Driver.

$ cat <<EOF | kubectl apply -f -
apiVersion: apps/v1
kind: DaemonSet
metadata:
  labels:
    app: csi-secrets-store-provider-vault
  name: csi-secrets-store-provider-vault
spec:
  updateStrategy:
    type: RollingUpdate
  selector:
    matchLabels:
      app: csi-secrets-store-provider-vault
  template:
    metadata:
      labels:
        app: csi-secrets-store-provider-vault
    spec:
      serviceAccount: secrets-store-csi-driver
      tolerations:
      containers:
        - name: provider-vault-installer
          image: hashicorp/secrets-store-csi-driver-provider-vault:0.0.4
          imagePullPolicy: Always
          resources:
            requests:
              cpu: 50m
              memory: 100Mi
            limits:
              cpu: 50m
              memory: 100Mi
          env:
            - name: TARGET_DIR
              value: "/etc/kubernetes/secrets-store-csi-providers"
          volumeMounts:
            - mountPath: "/etc/kubernetes/secrets-store-csi-providers"
              name: providervol
      volumes:
        - name: providervol
          hostPath:
              path: "/etc/kubernetes/secrets-store-csi-providers"
      nodeSelector:
        beta.kubernetes.io/os: linux
EOF
daemonset.apps/csi-secrets-store-provider-vault created

This DaemonSet launches its own provider pod with the name prefixed with csi-secrets-store-provider-vault and mounts the executable in the existing csi-secrets-store-csi-driver pod.

Get all the pods within the default namespace.

$ kubectl get pods
NAME                                     READY   STATUS    RESTARTS   AGE
csi-secrets-store-csi-driver-6rf2k       3/3     Running   0          4m40s
csi-secrets-store-provider-vault-qm44g   1/1     Running   0          20s
vault-0                                  1/1     Running   0          27m

Wait until csi-secrets-store-provider-vault pod is running and ready (1/1).

Verify that the provider-vault executable is present on the secrets-store container in the secrets-store-csi-driver pod.

$ kubectl exec \
  $(kubectl get pod -l app=secrets-store-csi-driver -o jsonpath="{.items[0].metadata.name}") \
  -c secrets-store -- \
  stat /etc/kubernetes/secrets-store-csi-providers/vault/provider-vault

  File: /etc/kubernetes/secrets-store-csi-providers/vault/provider-vault
  Size: 8861147     Blocks: 17312      IO Block: 4096   regular file
Device: 10h/16d Inode: 111064      Links: 1
Access: (0755/-rwxr-xr-x)  Uid: (    0/    root)   Gid: (    0/    root)
Access: 2020-03-19 20:25:20.726542817 +0000
Modify: 2020-03-19 20:25:20.825490265 +0000
Change: 2020-03-19 20:25:20.825490265 +0000
 Birth: -

The executable enables the Vault secrets provider when you define a SecretProviderClass.

»Define a SecretProviderClass resource

The Kubernetes-Secrets-Store-CSI-Driver Helm chart creates a definition for a SecretProviderClass resource. This resource describes the parameters that are given to the provider-vault executable. To configure it requires the IP address of the Vault server, the name of the Vault Kubernetes authentication role, and the secrets.

Create a SecretProviderClass named vault-database.

$ cat <<EOF | kubectl apply -f -
apiVersion: secrets-store.csi.x-k8s.io/v1alpha1
kind: SecretProviderClass
metadata:
  name: vault-database
spec:
  provider: vault
  parameters:
    vaultAddress: "http://vault.default:8200"
    roleName: "database"
    vaultSkipTLSVerify: "true"
    objects:  |
      array:
        - |
          objectPath: "/db-pass"
          objectName: "password"
          objectVersion: ""
EOF
secretproviderclass.secrets-store.csi.x-k8s.io/vault-database created

The vault-database SecretProviderClass describes one secret object:

  • objectPath is the path to the secret defined in Vault. Prefaced with a forward-slash.
  • objectName a key name within that secret
  • objectVersion - the version of the secret. When none is specified the latest is retrieved.

Verify that the SecretProviderClass, named vault-database has been defined in the default namespace.

$ kubectl describe SecretProviderClass vault-database
Name:         vault-database
Namespace:    default
## ...

»Create a pod with secret mounted

With the secret stored in Vault, the authentication configured and role created, the provider-vault extension installed and the SecretProviderClass defined it is finally time to create a pod that mounts the desired secret.

Apply the webapp pod that mounts the secrets volume.

$ cat <<EOF | kubectl apply -f -
kind: Pod
apiVersion: v1
metadata:
  name: webapp
spec:
  containers:
  - image: jweissig/app:0.0.1
    name: webapp
    volumeMounts:
    - name: secrets-store-inline
      mountPath: "/mnt/secrets-store"
      readOnly: true
  volumes:
    - name: secrets-store-inline
      csi:
        driver: secrets-store.csi.k8s.io
        readOnly: true
        volumeAttributes:
          secretProviderClass: "vault-database"
EOF
pod/webapp created

The webapp pod defines and mounts a read-only volume to /mnt/secrets-store. The objects defined in the vault-database SecretProviderClass are written as files within that path.

Get all the pods within the default namespace.

$ kubectl get pods
NAME                                     READY   STATUS    RESTARTS   AGE
csi-secrets-store-csi-driver-6rf2k       3/3     Running   0          13m
csi-secrets-store-provider-vault-qm44g   1/1     Running   0          8m
webapp                                   1/1     Running   0          5m
vault-0                                  1/1     Running   0          27m

Wait until the webapp pod is running and ready (1/1).

Display the password secret written to the file system at /mnt/secrets-store/db-pass on the webapp pod.

$ kubectl exec webapp -- cat /mnt/secrets-store/db-pass
db-secret-password

The value displayed matches the password value for the secret secret/db-pass.

»Next steps

The Kubernetes Container Storage Interface (CSI) is an extensible approach to the management of storage alongside the lifecycle of containers. Learn more about the Secrets Store CSI driver and the Vault provider in this guide to accomplish the secrets management for the container.

Secrets mounted on empheral volumes is one approach to manage secrets for applications pods. Explore how pods can retrieve them directly via network requests and through the Vault Injector service via annotations.