Run Vault on Kubernetes

Vault Installation to Minikube via Helm

Running Vault on Kubernetes is generally the same as running it anywhere else. Kubernetes, as a container orchestration engine, eases some of the operational burdens. Helm charts provide the benefit of a refined interface when it comes to deploying Vault in a variety of different modes.

In this guide, you will setup Vault and its dependencies with a Helm chart. Then integrate a web application that uses the Kubernetes service account token to authenticate with Vault and retrieve a secret.

Prerequisites

This guide requires the Kubernetes command-line interface (CLI) and the Helm CLI installed, Minikube, the Vault and Consul Helm charts for, the sample web application, and additional configuration to bring it all together.

This guide was last tested 23 Dec 2019 on a macOS 10.15.2 using this configuration:

$ docker version
Client: Docker Engine - Community
  Version:          19.03.5
  ...

$ minikube version
minikube version: v1.5.2
commit: 792dbf92a1de583fcee76f8791cff12e0c9440ad

$ helm version
Client: &version.Version{SemVer:"v2.16.1", GitCommit:"bbdfe5e7803a12bbdf97e94cd847859890cf4050", GitTreeState:"clean"}
# Tiller's version appears after `helm init` later in the guide.
Error: could not find tiller

Although we recommend these software versions, the output you see may vary depending on your environment and the software versions you use.

First, follow the directions for installing Minikube, including VirtualBox or similar.

Next, install kubectl CLI and helm CLI.

On Mac with Homebrew.

$ brew install kubernetes-cli
$ brew install helm@2

On Windows with Chocolatey:

$ choco install kubernetes-cli
$ choco install kubernetes-helm --version 2.16.0

Next, retrieve the web application and additional configuration by cloning the hashicorp/vault-guides repository from GitHub.

$ git clone https://github.com/hashicorp/vault-guides.git

This repository contains supporting content for all of the Vault learn guides. The content specific to this guide can be found within a sub-directory.

Go into the vault-guides/operations/provision-vault/kubernetes/minikube/getting-started directory.

$ cd vault-guides/operations/provision-vault/kubernetes/minikube/getting-started

Start Minikube

Minikube is a CLI tool that provisions and manages the lifecycle of single-node Kubernetes clusters locally inside Virtual Machines (VM) on your system.

Start a Kubernetes cluster with 4096 Megabytes (MB) of memory:

$ minikube start --memory 4096
🙄  minikube v1.5.2 on Darwin 10.15.2
✨  Automatically selected the 'hyperkit' driver (alternates: [virtualbox])
🔥  Creating hyperkit VM (CPUs=2, Memory=4096MB, Disk=20000MB) ...
🐳  Preparing Kubernetes v1.16.2 on Docker '18.09.9' ...
🚜  Pulling images ...
🚀  Launching Kubernetes ...
⌛  Waiting for: apiserver
🏄  Done! kubectl is now configured to use "minikube"

The --memory is set to 4096 MB to ensure there is plenty of memory for all the resources to run concurrently. The initialization process takes several minutes as it retrieves any necessary dependencies and executes various container images.

Verify the status of the Minikube cluster:

$ minikube status
host: Running
kubelet: Running
apiserver: Running
kubeconfig: Configured

The host, kubelet, apiserver report that they are running. The kubectl, a command line interface (CLI) for running commands against Kubernetes cluster, is also configured to communicate with this recently started cluster.

Minikube provides a visual representation of the status in a web-based dashboard. This interface makes it easy to view cluster activity and delve into the issues affecting it.

In another terminal, launch the minikube dashboard:

$ minikube dashboard

The operating system's default browser opens and displays the dashboard.

Minikube Dashboard

Initialize Helm

Helm is a package manager that provides the community the ability to share charts that describe the operation of resources operating within Kubernetes. Helm requires initialization so that it can load an in-cluster component called Tiller. Tiller interacts directly with the Kubernetes API Server to install, upgrade, query, and remove Kubernetes resources.

Initialize Helm and start Tiller:

$ helm init
$HELM_HOME has been configured at $HOME/.helm.

Tiller (the Helm server-side component) has been installed into your Kubernetes Cluster.

Please note: by default, Tiller is deployed with an insecure 'allow unauthenticated users' policy.
To prevent this, run `helm init` with the --tiller-tls-verify flag.
For more information on securing your installation see: https://docs.helm.sh/using_helm/#securing-your-helm-installation

Tiller runs as a pod within the kube-system namespace. A pod is a Kubernetes resource that represents one or more cooperating processes that form a cohesive unit of service. Namespaces, another Kubernetes resource, provide a scope for names. The kube-system is the namespace specifically for resources created to support the Kubernetes system.

Verify that Tiller is running by getting all the pods within the kube-system namespace:

$ kubectl get pods --namespace kube-system
NAME                                    READY   STATUS    RESTARTS   AGE
coredns-5c98db65d4-s8cdv                1/1     Running   1          7m17s
coredns-5c98db65d4-vh5tw                1/1     Running   1          7m17s
etcd-minikube                           1/1     Running   0          6m20s
kube-addon-manager-minikube             1/1     Running   0          6m19s
kube-apiserver-minikube                 1/1     Running   0          6m12s
kube-controller-manager-minikube        1/1     Running   0          6m9s
kube-proxy-llgmm                        1/1     Running   0          7m17s
kube-scheduler-minikube                 1/1     Running   0          6m12s
kubernetes-dashboard-7b8ddcb5d6-7gs2l   1/1     Running   0          7m16s
storage-provisioner                     1/1     Running   0          7m16s
tiller-deploy-75f6c87b87-n4db8          1/1     Running   0          21s

The Tiller service appears here as the pod named tiller-deploy-75f6c87b87-n4db8. It reports that it is ready and running.

Verify that Tiller is running, through the dashboard, by getting all the pods within the kube-system namespace:

Minikube dashboard highlighting the "kube-system" namespace

Install the Consul Helm chart

Consul is a service mesh solution that launches with a key-value store. Vault requires a storage backend to manage its configuration and secrets. The Vault Helm chart defaults to define a Consul storage backend.

The recommended way to run Consul on Kubernetes is via the Helm chart. This installs and configures all the necessary components to run Consul in several different modes. A Helm chart includes templates that enable conditional and parameterized execution. These parameters can be set through command-line arguments or defined in YAML. For Consul to act as a storage backend for Vault requires a minimum of a server and a client.

$ cat helm-consul-values.yml
global:
  datacenter: vault-kubernetes-guide

client:
  enabled: true

server:
  replicas: 1
  bootstrapExpect: 1
  disruptionBudget:
    maxUnavailable: 0

Install the Consul Helm chart version 0.15.0 with pods prefixed with the name consul and apply the values found in helm-consul-values.yml:

$ helm install --name consul \
    --values helm-consul-values.yml \
    https://github.com/hashicorp/consul-helm/archive/v0.15.0.tar.gz

NAME:   consul
LAST DEPLOYED: Mon Dec 23 16:31:00 2019
NAMESPACE: default
STATUS: DEPLOYED

RESOURCES:
...


NOTES:
...

Your release is named consul. To learn more about the release, try:

  $ helm status consul
  $ helm get consul

The installation of the Helm chart displays the namespace, status, and resources created. The server and client pods were deployed in the default namespace because no namespace was specified or configured as the default.

To verify, get all the pods within the default namespace:

$ kubectl get pods
NAME                     READY   STATUS    RESTARTS   AGE
consul-consul-6jcfj      1/1     Running   0          2m54s
consul-consul-server-0   1/1     Running   0          2m54s

The Consul services appears here as the pods prefixed with consul. The server and client report that they are Running and ready (1/1).

Minikube dashboard showing Consul pods

For more information refer to the Installing Consul to Minikube via Helm guide

Install the Vault Helm chart

The recommended way to run Vault on Kubernetes is via the Helm chart. This installs and configures all the necessary components to run Vault in several different modes.

Install the Vault Helm chart version 0.3.0 with pods prefixed with the name vault:

$ helm install --name vault \
    https://github.com/hashicorp/vault-helm/archive/v0.3.0.tar.gz

NAME:   vault
LAST DEPLOYED: Mon Dec 23 16:35:28 2019
NAMESPACE: default
STATUS: DEPLOYED

RESOURCES:
...

NOTES:
...

Your release is named vault. To learn more about the release, try:

  $ helm status vault
  $ helm get vault

The Vault pod is deployed in the default namespace.

To verify, get all the pods within the default namespace:

$ kubectl get pods
NAME                                    READY   STATUS    RESTARTS   AGE
...
vault-0                                 0/1     Running   0          2m41s
vault-agent-injector-5945fb98b5-thczv   1/1     Running   0          2m42s

The vault-0 pod reports that it is Running but it is not ready 0/1. The Helm chart defines a readinessProbe that executes a command to check Vault's status.

Retrieve the status of Vault on the vault-0 pod:

$ kubectl exec vault-0 -- vault status
Key                Value
---                -----
Seal Type          shamir
Initialized        false
Sealed             true
Total Shares       0
Threshold          0
Unseal Progress    0/0
Unseal Nonce       n/a
Version            n/a
HA Enabled         false
command terminated with exit code 2

The status command reports that Vault is not initialized and that it is still sealed.

Minikube dashboard showing sealed Vault pod

The web interface reports the same results.

Initialize and unseal Vault

When Vault is started, it starts uninitialized and in the sealed state. Prior to initialization the storage backend, Consul, is not prepared to receive data.

Start an interactive shell session on the vault-0 pod:

$ kubectl exec -it vault-0 -- /bin/sh
/ $

Your system prompt is replaced with a new prompt / $. Commands issued at this prompt are executed on the vault-0 container.

Initialize Vault with one key share and one key threshold:

/ $ vault operator init -key-shares 1 -key-threshold 1

Unseal Key 1: HrQuCtWWqQyhZ/AT5JHXM8n1e5tJjiv3mM8Jsg7TGwQ=

Initial Root Token: s.GHXCSwHq3BzDDs7Zx0baPPv1
...

The init command generates a master key that it disassembles into key shares and then sets the number of key shares required to unseal Vault. These key shares appear in the output as unseal keys.

The command also generated an initial root token which is used in the next step to login to Vault.

After initialization, Vault is configured to know where and how to access the storage, but doesn't know how to decrypt any of it. Unsealing is the process of constructing the master key necessary to read the decryption key to decrypt the data, allowing access to the Vault.

Unseal Vault with Unseal Key 1:

/ $ vault operator unseal HrQuCtWWqQyhZ/AT5JHXM8n1e5tJjiv3mM8Jsg7TGwQ=
Key             Value
---             -----
Seal Type       shamir
Initialized     true
Sealed          false
Total Shares    1
Threshold       1
Version         1.3.1
Cluster Name    vault-cluster-802b70ee
Cluster ID      c71ef6a5-b0b1-ee7d-f4c8-3a2df9cf911a
HA Enabled      false

The unseal command reports that Vault is initialized and unsealed.

Vault is now ready for you to login with the initial root token.

Vault requires clients to authenticate first before it allows any further actions. An unsealed Vault starts with the Token Auth Method enabled and generates an initial root token.

Finally, login with the root token:

/ $ vault login s.GHXCSwHq3BzDDs7Zx0baPPv1
Success! You are now authenticated. The token information displayed below
is already stored in the token helper. You do NOT need to run "vault login"
again. Future Vault requests will automatically use this token.

Key                  Value
---                  -----
token                s.GHXCSwHq3BzDDs7Zx0baPPv1
token_accessor       JVsMJHVu6rTWbPLlYmWQTq1R
token_duration       ∞
token_renewable      false
token_policies       ["root"]
identity_policies    []
policies             ["root"]

Set a secret in Vault

The web application that you deploy in the final step, expects Vault to store a username and password stored at the path secret/exampleapp/config. To create this secret requires that a key-value secret engine is enabled and a username and password is put at the specified path.

Enable kv-v2 secrets at the path secret:

/ $ vault secrets enable -path=secret kv-v2
Success! Enabled the kv-v2 secrets engine at: secret/

Put a username and password secret at the path secret/exampleapp/config:

/ $ vault kv put secret/exampleapp/config username="helmchart" password="secrets"
Key              Value
---              -----
created_time     2019-12-23T22:50:04.812795276Z
deletion_time    n/a
destroyed        false
version          1

Verify that the secret is defined at the path secret/exampleapp/config:

/ $ vault kv get secret/exampleapp/config
====== Metadata ======
Key              Value
---              -----
created_time     2019-12-23T22:50:04.812795276Z
deletion_time    n/a
destroyed        false
version          1

====== Data ======
Key         Value
---         -----
password    secrets
username    helmchart

You created a secret for the web application. Now the web application needs to authenticate via Kubernetes and granted a policy to read this secret.

Configure Kubernetes authentication

The initial root token is a privileged user that can perform any operation and this web application only requires the ability to read a single secret. A policy can define a smaller set of capabilities and these policies are assigned to an authentication method.

Vault provides a Kubernetes authentication method that enables clients to authenticate with a Kubernetes Service Account Token.

Enable the Kubernetes authentication method:

/ $ vault auth enable kubernetes
Success! Enabled kubernetes auth method at: kubernetes/

Vault accepts this service token from any client within the Kubernetes cluster. During authentication, Vault verifies that the service account token is valid by querying a configured Kubernetes endpoint.

Configure the Kubernetes authentication method to use the service account token, the location of the Kubernetes host, and its certificate:

/ $ vault write auth/kubernetes/config \
        token_reviewer_jwt="$(cat /var/run/secrets/kubernetes.io/serviceaccount/token)" \
        kubernetes_host="https://$KUBERNETES_PORT_443_TCP_ADDR:443" \
        kubernetes_ca_cert=@/var/run/secrets/kubernetes.io/serviceaccount/ca.crt
Success! Data written to: auth/kubernetes/config

The token_reviewer_jwt and kubernetes_ca_cert are files written to the container by Kubernetes. The environment variable KUBERNETES_PORT_443_TCP_ADDR references the internal network address of the Kubernetes host.

For a client to access the secret data defined, at secret/exampleapp/config, requires that the read capability be granted for the path secret/data/exampleapp/config.

Write out the policy named exampleapp that enables the read capability for secrets at path secret/data/exampleapp/config

/ $ vault policy write exampleapp - <<EOH
path "secret/data/exampleapp/config" {
  capabilities = ["read"]
}
EOH
Success! Uploaded policy: exampleapp

Create a Kubernetes authentication role, named exampleapp, that connects the Kubernetes service account name and policy:

$ vault write auth/kubernetes/role/exampleapp \
        bound_service_account_names=vault \
        bound_service_account_namespaces=default \
        policies=exampleapp \
        ttl=24h
Success! Data written to: auth/kubernetes/role/exampleapp

The role connects the Kubernetes service account, vault, and namespace, default, with the Vault policy, exampleapp. The tokens returned after authentication are valid for 24 hours.

Lastly, exit the the vault-0 pod:

/ $ exit
$

Launch a web application

We've created a web application, published it to DockerHub, and created a Kubernetes manifest that will run the application in your existing cluster. The example web application performs the single function of listening for requests on port 8080. During a request it reads the Kubernetes service token, logs into Vault, and then requests the secret.

Deploy the exampleapp in Kubernetes by applying the file deployment-01-exampleapp.yml:

$ kubectl apply --filename deployment-01-exampleapp.yml

The example application runs as a pod within the default namespace.

Get all the pods within the default namespace:

$ kubectl get pods
NAME                                    READY   STATUS    RESTARTS   AGE
consul-consul-6jcfj                     1/1     Running   0          43m
consul-consul-server-0                  1/1     Running   0          43m
exampleapp-5c76d96c6-r4mcq              1/1     Running   0          2m43s
vault-0                                 1/1     Running   0          38m
vault-agent-injector-5945fb98b5-thczv   1/1     Running   0          38m

The example service appears here as the pod named exampleapp-5c76d96c6-r4mcq.

This web application, running in the exampleapp-5c76d96c6-r4mcq pod, is running an HTTP service that is listening on port 8080.

In another terminal, port forward all requests made to http://localhost:8080 to the exampleapp pod on port 8200:

$ kubectl port-forward exampleapp-5c76d96c6-r4mcq 8080:8080

In the original terminal, execute a curl request at http://localhost:8080:

$ curl http://localhost:8080
{"password"=>"secret", "username"=>"helmchart"}%

Web application showing username and password secret

The web application running on port 8080 in the exampleapp pod:

  • authenticates with the Kubernetes service account token
  • receives a token with the read capability at the secret/data/exampleapp/config path
  • retrieves the secrets from secret/data/exampleapp/config path
  • displays the secrets as JSON

Summary

For more on Consul's integration with Kubernetes (including multi-cloud, service sync, and other features), see the Consul with Kubernetes documentation. The Consul and Vault Helm charts provide configuration settings that can be explored in their respective repositories.