Kubernetes application pods that rely on Vault to manage their secrets can retrieve them directly via network requests or maintained on a mounted file system through the Vault Injector service via annotations or attached as ephemeral volumes. This approach of employing ephemeral volumes to store secrets is a feature of the Secrets Store extension to the Kubernetes Container Storage Interface (CSI) driver.
In this tutorial, you will setup Vault and its dependencies with a Helm chart. Then enable and configure the secrets store CSI driver to create a volume that contains a secret that you will mount to an application pod.
»Prerequisites
This tutorial requires the Kubernetes command-line interface (CLI) and the Helm CLI installed, Minikube, and additional configuration to bring it all together.
Online tutorial: An interactive tutorial is also available if you do not wish to install the following resources. Click the Show Terminal button to start.
This tutorial was last tested 30 Mar 2021 on macOS 10.15.7 using this configuration.
Docker version.
Minikube version.
Helm version.
These are recommended software versions and the output displayed may vary depending on your environment and the software versions you use.
First, follow the directions to install Minikube, including VirtualBox or similar.
Next, install kubectl CLI and helm CLI.
»Start Minikube
Minikube is a CLI tool that provisions and manages the lifecycle of single-node Kubernetes clusters. These clusters are run locally inside Virtual Machines (VM).
Start a Kubernetes cluster.
Display the version of the Kubernetes cluster.
Kubernetes version 1.19.0 and lower requires a local Kubernetes cluster started with two additional arguments to set the apiserver service account credentials.
Stop the cluster.
Start a Kubernetes cluster with additional arguments.
Verify the status of the Minikube cluster.
Additional waiting: Even if this last command completed successfully, you may have to wait for Minikube to be available. If an error is displayed, try again after a few minutes.
The host, kubelet, apiserver report that they are running. The kubectl, a
command line interface (CLI) for running commands against Kubernetes cluster, is
also configured to communicate with this recently started cluster.
»Install the Vault Helm chart
Vault manages the secrets that are written to these mountable volumes. To provide these secrets a single Vault server is required. For this demonstration Vault can be run in development mode to automatically handle initialization, unsealing, and setup of a KV secrets engine.
Add the HashiCorp Helm repository.
Update all the repositories to ensure helm is aware of the latest versions.
Install the latest version of the Vault Helm chart running in development mode with the injector service disabled and CSI enabled.
The Vault server runs in development mode on a single pod
server.dev.enabled=true. The Vault Agent Injector pod is disabled
injector.enabled=false and the Vault CSI Provider pod csi.enabled=true is
enabled.
Display all the pods within the default namespace.
Wait until the vault-0 pod is running and ready (1/1).
»Set a secret in Vault
The volume mounted to the pod in the Create a pod with secret
mounted section expects a secret stored at
the path secret/data/db-pass. When Vault is run in development a KV
secret engine is
enabled at the path /secret.
First, start an interactive shell session on the vault-0 pod.
Your system prompt is replaced with a new prompt / $. Commands issued at this
prompt are executed on the vault-0 container.
Create a secret at the path secret/db-pass with a password.
Verify that the secret is readable at the path secret/db-pass.
Lastly, exit the vault-0 pod.
»Configure Kubernetes authentication
Vault provides a Kubernetes authentication method that enables clients to authenticate with a Kubernetes Service Account Token. The Kubernetes resources that access the secret and create the volume authenticate through this method through a role.
First, start an interactive shell session on the vault-0 pod.
Your system prompt is replaced with a new prompt / $. Commands issued at this
prompt are executed on the vault-0 container.
Enable the Kubernetes authentication method.
Configure the Kubernetes authentication method to use the service account token, the location of the Kubernetes host, and its certificate.
The token_reviewer_jwt and kubernetes_ca_cert files written to the container
by Kubernetes. The environment variable KUBERNETES_PORT_443_TCP_ADDR
references the internal network address of the Kubernetes host.
For the Kubernetes-Secrets-Store-CSI-Driver to read the secrets requires that it has read permissions of all mounts and access to the secret itself.
Write out the policy named internal-app.
The data of
kv-v2 requires that
an additional path element of data is included after its mount path (in this
case, secret/).
Finally, create a Kubernetes authentication role named database that binds
this policy with a Kubernetes service account named webapp-sa.
The role connects the Kubernetes service account, webapp-sa, in
the namespace, default, with the Vault policy, internal-app. The tokens
returned after authentication are valid for 20 minutes. This Kubernetes service
account name, webapp-sa, will be created below.
Lastly, exit the vault-0 pod.
»Install the secrets store CSI driver
The Secrets Store CSI driver secrets-store.csi.k8s.io allows Kubernetes to mount multiple secrets, keys, and certs stored in enterprise-grade external secrets stores into their pods as a volume. Once the Volume is attached, the data in it is mounted into the container's file system.
Add the Secrets Store CSI driver Helm repository.
Install the latest version of the Kubernetes-Secrets-Store-CSI-Driver.
The csi-secrets-store-csi-driver pod is deployed in the default
namespace.
Get all the pods within the default namespace.
Wait until the csi-secrets-store-csi-driver pod is running and ready
(3/3).
»Verify the Vault CSI provider is running
The Secrets Store CSI driver enables extension through providers. A provider is launched as a Kubernetes DaemonSet alongside of Secrets Store CSI driver DaemonSet.
The Vault CSI provider was installed above alongside Vault by the Vault Helm chart.
This DaemonSet launches its own provider pod and runs a gRPC server which the Secrets Store CSI Driver connects to to make volume mount requests.
Get all the pods within the default namespace to check that the Vault CSI provider is running.
Wait until the vault-csi-provider pod is running and ready (1/1).
»Define a SecretProviderClass resource
The Kubernetes-Secrets-Store-CSI-Driver Helm chart creates a definition for a SecretProviderClass resource. This resource describes the parameters that are given to the Vault CSI provider. To configure it requires the address of the Vault server, the name of the Vault Kubernetes authentication role, and the secrets.
Define a SecretProviderClass named vault-database.
Create the vault-database SecretProviderClass.
The vault-database SecretProviderClass describes one secret object:
objectNameis a symbolic name for that secret, and the file name to write to.secretPathis the path to the secret defined in Vault.secretKeyis a key name within that secret.
Verify that the SecretProviderClass, named vault-database has been defined in
the default namespace.
»Create a pod with secret mounted
With the secret stored in Vault, the authentication configured and role created,
the provider-vault extension installed and the SecretProviderClass defined
it is finally time to create a pod that mounts the desired secret.
Define the webapp-sa service account and the webapp pod that mounts the
secrets volume.
The webapp pod defines and mounts a read-only volume to /mnt/secrets-store.
The objects defined in the vault-database SecretProviderClass are written as
files within that path.
Create the webapp-sa service account and webapp pod.
Get all the pods within the default namespace.
Wait until the webapp pod is running and ready (1/1).
Display the password secret written to the file system at
/mnt/secrets-store/db-password on the webapp pod.
The value displayed matches the password value for the secret
secret/db-pass.
»Next steps
The Kubernetes Container Storage Interface (CSI) is an extensible approach to the management of storage alongside the lifecycle of containers. Learn more about the Secrets Store CSI driver and the Vault provider in this tutorial to accomplish the secrets management for the container.
Secrets mounted on ephemeral volumes is one approach to manage secrets for applications pods. Explore how pods can retrieve them directly via network requests and through the Vault Injector service via annotations.
