Nearly all requests to Vault must be accompanied by an authentication token. This includes all API requests, as well as via the Vault CLI and other libraries. If you can securely get the first secret from an originator to a consumer, all subsequent secrets transmitted between this originator and consumer can be authenticated with the trust established by the successful distribution and user of that first secret.
»Challenge
The applications running in a Kubernetes environment is no exception. Luckily, Vault provides Kubernetes auth method to authenticate the clients using a Kubernetes Service Account Token.
However, the client is still responsible for managing the lifecycle of its Vault tokens. Therefore, the next challenge becomes how to manage the lifecycle of tokens in a standard way without having to write custom logic.
»Solution
Vault Agent provides a number of different helper features, specifically addressing the following challenges:
- Automatic authentication
- Secure delivery/storage of tokens
- Lifecycle management of these tokens (renewal & re-authentication)
This tutorial focuses on the demonstration of the Vault Agent Auto-Auth using
the kubernetes
auth method. Vault Helm introduced Agent Sidecar
Injector. Refer to
Injecting Secrets into Kubernetes Pods via Vault Helm
Sidecar for a step-by-step tutorial on the
sidecar usage.
»Prerequisites
To perform the tasks described in this tutorial, you need:
- Minikube installed
- A running Vault environment reachable from your Kubernetes environment. Refer to the Getting Started tutorial to install Vault. Make sure that your Vault server has been initialized and unsealed
NOTE: For the purpose of demonstration, this tutorial runs Minikube as a Kubernetes environment. If you wish to test against an Azure Kubernetes Service (AKS) cluster instead, follow the steps in the Azure Kubernetes Service Cluster section to create an AKS cluster. If you want to test against a Google Kubernetes Engine (GKE) cluster, refer to the steps described in the Google Kubernetes Engine Cluster section.
»Download demo assets
Clone or download the demo assets from the hashicorp/vault-guides GitHub repository to perform the steps described in this tutorial.
»Start a Vault server
To go through this tutorial, start a Vault dev server which listens for requests
locally at 0.0.0.0:8200
with root
as the root token ID.
$ vault server -dev -dev-root-token-id root -dev-listen-address 0.0.0.0:8200
Setting the -dev-listen-address
to 0.0.0.0:8200
overrides the default
address of a Vault dev server (127.0.0.1:8200
) and enables Vault to be
addressable by the Kubernetes cluster and its pods because it binds to a
shared network.
Insecure operation: Do not run a Vault dev server in production. This approach is only used here to simplify the unsealing process for this demonstration.
Export an environment variable for the vault
CLI to address the Vault server.
$ export VAULT_ADDR=http://0.0.0.0:8200
»Step 1: Create a service account
Start a Kubernetes cluster running in Minikube.
$ minikube start
Wait a couple of minutes for the minikube environment to become fully available.
$ minikube status host: Running kubelet: Running apiserver: Running kubeconfig: Configured
From a command terminal, set your working directory to where the
/identity/vault-agent-k8s-demo
folder is located. The working directory should contain 2 directories and 14 files.$ cd vault-guides/identity/vault-agent-k8s-demo $ tree . ├── README.md ├── configmap.yaml ├── example-k8s-spec.yaml ├── setup-k8s-auth.sh ├── terraform-azure │ ├── README.md │ ├── aks-cluster.tf │ ├── terraform.tfvars.example │ └── versions.tf ├── terraform-gcp │ ├── README.md │ ├── main.tf │ ├── terraform.tfvars.example │ ├── variables.tf │ └── versions.tf └── vault-auth-service-account.yaml 2 directories, 14 files
In Kubernetes, a service account provides an identity for processes that run in a Pod so that the processes can contact the API server. Examine the provided
vault-auth-service-account.yaml
file for the service account definition to be used for this tutorial.$ cat vault-auth-service-account.yaml --- apiVersion: rbac.authorization.k8s.io/v1beta1 kind: ClusterRoleBinding metadata: name: role-tokenreview-binding namespace: default roleRef: apiGroup: rbac.authorization.k8s.io kind: ClusterRole name: system:auth-delegator subjects: - kind: ServiceAccount name: vault-auth namespace: default
Create a Kubernetes service account named
vault-auth
.$ kubectl create serviceaccount vault-auth
Update the
vault-auth
service account.$ kubectl apply --filename vault-auth-service-account.yaml
»Step 2: Configure Kubernetes auth method
Create a read-only policy,
myapp-kv-ro
in Vault.$ vault policy write myapp-kv-ro - <<EOF path "secret/data/myapp/*" { capabilities = ["read", "list"] } EOF
Create some test data at the
secret/myapp
path.$ vault kv put secret/myapp/config username='appuser' \ password='suP3rsec(et!' \ ttl='30s'
Set the environment variables to point to the running Minikube environment.
Set the
VAULT_SA_NAME
environment variable value to the service account you created earlier.$ export VAULT_SA_NAME=$(kubectl get sa vault-auth \ -o jsonpath="{.secrets[*]['name']}")
Set the
SA_JWT_TOKEN
environment variable value to the service account JWT used to access the TokenReview API$ export SA_JWT_TOKEN=$(kubectl get secret $VAULT_SA_NAME \ -o jsonpath="{.data.token}" | base64 --decode; echo)
Set the
SA_CA_CRT
environment variable value to the PEM encoded CA cert used to talk to Kubernetes API.$ export SA_CA_CRT=$(kubectl get secret $VAULT_SA_NAME \ -o jsonpath="{.data['ca\.crt']}" | base64 --decode; echo)
Set the
K8S_HOST
environment variable value to minikube IP address.$ export K8S_HOST=$(minikube ip)
Now, enable and configure the Kubernetes auth method.
Enable the Kubernetes auth method at the default path ("auth/kubernetes").
$ vault auth enable kubernetes
Tell Vault how to communicate with the Kubernetes (Minikube) cluster.
$ vault write auth/kubernetes/config \ token_reviewer_jwt="$SA_JWT_TOKEN" \ kubernetes_host="https://$K8S_HOST:8443" \ kubernetes_ca_cert="$SA_CA_CRT"
Create a role named, "example" to map Kubernetes Service Account to Vault policies and default token TTL.
$ vault write auth/kubernetes/role/example \ bound_service_account_names=vault-auth \ bound_service_account_namespaces=default \ policies=myapp-kv-ro \ ttl=24h
NOTE: The pattern Vault uses to authenticate Pods depends on sharing the JWT token over the network. Given the security model of Vault, this is allowable because Vault is part of the trusted compute base. In general, Kubernetes applications should not share this JWT with other applications, as it allows API calls to be made on behalf of the Pod and can result in unintended access being granted to 3rd parties.
»Step 3: Determine the Vault address
A service bound to all networks on the host, as you configured Vault, is addressable by pods within Minikube's cluster by sending requests to the gateway address of the Kubernetes cluster.
Start a minikube SSH session.
$ minikube ssh ## ... minikube ssh login
Within this SSH session, retrieve the value of the Minikube host.
$ route -n | grep ^0.0.0.0 | awk '{ print $2 }' 192.168.64.1
In this example, the Vault address is http://192.168.64.1:8200
.
Verify the connectivity to the Vault server using the address you discovered.
$ curl -s http://192.168.64.1:8200/v1/sys/seal-status | jq { "type": "shamir", "initialized": true, "sealed": false, "t": 1, "n": 1, "progress": 0, "nonce": "", "version": "1.4.1+ent", "migration": false, "cluster_name": "vault-cluster-3de6c2d3", "cluster_id": "10fd177e-d55a-d740-0c54-26268ed86e31", "recovery_seal": false, "storage_type": "inmem" }
The output displays that Vault is initialized and unsealed. This confirms that pods within your cluster are able to reach Vault given that each pod is configured to use the gateway address.
Exit the Minikube SSH session.
$ exit
Finally, create a variable named
EXTERNAL_VAULT_ADDR
to capture the Minikube gateway address.Example:
$ EXTERNAL_VAULT_ADDR="http://192.168.64.1:8200"
»Optional: Verify the Kubernetes auth method configuration
Create a Pod with a container running
alpine:3.7
image.$ kubectl run --generator=run-pod/v1 tmp --rm -i --tty \ --serviceaccount=vault-auth --image alpine:3.7
Once you are inside the container, install cURL and jq tools.
/# apk update /# apk add curl jq
Set
KUBE_TOKEN
to the service account token value./# KUBE_TOKEN=$(cat /var/run/secrets/kubernetes.io/serviceaccount/token) /# echo $KUBE_TOKEN
Authenticate with Vault using the Vault address you discovered earlier and set to
EXTERNAL_VAULT_ADDR
. In the following example, the value ishttp://192.168.64.1:8200
./# curl --request POST \ --data '{"jwt": "'"$KUBE_TOKEN"'", "role": "example"}' \ http://192.168.64.1:8200/v1/auth/kubernetes/login | jq { ... "auth": { "client_token": "s.7cH83AFIdmXXYKsPsSbeESpp", "accessor": "8bmYWFW5HtwDHLAoxSiuMZRh", "policies": [ "default", "myapp-kv-ro" ], "token_policies": [ "default", "myapp-kv-ro" ], "metadata": { "role": "example", "service_account_name": "vault-auth", "service_account_namespace": "default", "service_account_secret_name": "vault-auth-token-vqqlp", "service_account_uid": "adaca842-f2a7-11e8-831e-080027b85b6a" }, "lease_duration": 86400, "renewable": true, "entity_id": "2c4624f1-29d6-972a-fb27-729b50dd05e2", "token_type": "service" } }
Notice that
client_token
is successfully generated andmyapp-kv-ro
policy is attached with the token. Themetadata
displays that its service account name (service_account_name
) isvault-auth
.Lastly, exit the pod.
/# exit
»Step 4: Leverage Vault Agent Auto-Auth
Now that you have verified that the Kubernetes auth method has been configured on the Vault server, it is time to spin up a client Pod which leverages Vault Agent to automatically authenticate with Vault and retrieve a client token.
First, review the provided
configmap.yaml
file.$ cat configmap.yaml apiVersion: v1 data: vault-agent-config.hcl: | # Comment this out if running as sidecar instead of initContainer exit_after_auth = true pid_file = "/home/vault/pidfile" auto_auth { method "kubernetes" { mount_path = "auth/kubernetes" config = { role = "example" } } sink "file" { config = { path = "/home/vault/.vault-token" } } } template { destination = "/etc/secrets/index.html" contents = <<EOT <html> <body> <p>Some secrets:</p> {{- with secret "secret/data/myapp/config" }} <ul> <li><pre>username: {{ .Data.data.username }}</pre></li> <li><pre>password: {{ .Data.data.password }}</pre></li> </ul> {{ end }} </body> </html> EOT } kind: ConfigMap metadata: name: example-vault-agent-config namespace: default
This creates a Vault Agent configuration file,
vault-agent-config.hcl
. Notice that the Vault Agent Auto-Auth (auto_auth
block) is configured to use thekubernetes
auth method enabled at theauth/kubernetes
path on the Vault server. The Vault Agent will use theexample
role which you created in Step 2.The
sink
block specifies the location on disk where to write tokens. Vault Agent Auto-Authsink
can be configured multiple times if you want Vault Agent to place the token into multiple locations. In this example, thesink
is set to/home/vault/.vault-token
.Finally, the
template
block creates a templated file which retrievesusername
andpassword
values at thesecret/data/myapp/config
path.Create a ConfigMap containing a Vault Agent configuration.
$ kubectl create -f configmap.yaml
View the created ConfigMap.
$ kubectl get configmap example-vault-agent-config -o yaml
An example Pod spec file is provided. Review the provided example Pod spec file,
example-k8s-spec.yaml
.$ cat example-k8s-spec.yaml apiVersion: v1 kind: Pod metadata: name: vault-agent-example namespace: default spec: serviceAccountName: vault-auth volumes: - configMap: items: - key: vault-agent-config.hcl path: vault-agent-config.hcl name: example-vault-agent-config name: config - emptyDir: {} name: shared-data initContainers: - args: - agent - -config=/etc/vault/vault-agent-config.hcl - -log-level=debug env: - name: VAULT_ADDR value: http://192.168.64.1:8200 image: vault name: vault-agent volumeMounts: - mountPath: /etc/vault name: config - mountPath: /etc/secrets name: shared-data containers: - image: nginx name: nginx-container ports: - containerPort: 80 volumeMounts: - mountPath: /usr/share/nginx/html name: shared-data
The example Pod spec (
example-k8s-spec.yaml
) spins up two containers invault-agent-example
Pod. Avault
container which runs Vault Agent as an Init Container. And annginx
container exposing port 80.NOTE: Set the
VAULT_ADDR
value to where your Vault server is running. If you are following the Minikube steps, this value should be theEXTERNAL_VAULT_ADDR
value you discovered in Step 3.Create the
vault-agent-example
Pod defined inexample-k8s-spec.yaml
.$ kubectl apply -f example-k8s-spec.yaml --record
This takes a minute or so for the Pod to become fully up and running.
»Verification
In another terminal, launch the Minikube dashboard.
$ minikube dashboard
Click Pods under Workloads to verify that
vault-agent-example
Pod has been created successfully.- Select vault-agent-example to see its details.
Port-forward so you can connect to the client from browser.
$ kubectl port-forward pod/vault-agent-example 8080:80
In a web browser, go to
localhost:8080
Notice that the
username
andpassword
values were successfully read fromsecret/myapp/config
.Optionally, you can view the HTML source.
$ kubectl exec -it vault-agent-example --container nginx-container sh /# cat /usr/share/nginx/html/index.html <html> <body> <p>Some secrets:</p> <ul> <li><pre>username: appuser</pre></li> <li><pre>password: suP3rsec(et!</pre></li> </ul> </body> </html>
»Azure Kubernetes Service Cluster
If you wish to test the Kubernetes auth method against an Azure Kubernetes Service (AKS) cluster instead of Minikube, you can run Terraform to provision an AKS cluster.
First, follow the instruction in the Terraform documentation to create a service principal.
Be sure to add the following Azure Active Directory Graph API permission to your app.
Store the credentials as Environment Variables.
$ export ARM_CLIENT_ID="00000000-0000-0000-0000-000000000000"
$ export ARM_CLIENT_SECRET="00000000-0000-0000-0000-000000000000"
$ export ARM_SUBSCRIPTION_ID="00000000-0000-0000-0000-000000000000"
$ export ARM_TENANT_ID="00000000-0000-0000-0000-000000000000"
Subscription ID: Navigate to the Subscriptions blade within the Azure Portal and copy the SUBSCRIPTION ID
Tenant ID: Navigate to the Azure Active Directory > Properties in the Azure Portal, and copy the Directory ID which is your tenant ID
Client ID: Same as the Application ID
Client secret: The password (credential) set on your application
Set your working directory to where the
/identity/vault-agent-k8s-demo/terraform-azure
folder is located.Modify
terraform.tfvars.example
and provide Azure credentials:client_id
andclient_secret
, and save it asterraform.tfvars
.
- Client ID: Same as the Application ID
- Client secret: The password (credential) set on your application
Execute the Terraform commands to provision a new AKS cluster.
Pull necessary plugins.
$ terraform init
Now, execute the
apply
command to build a new AKS cluster.$ terraform apply -auto-approve
NOTE: If you received
Resource group 'education-XXXXX-rg' could not be found
error, runterraform refresh
command.$ terraform refresh ... Outputs: kubernetes_cluster_name = education-xxxxxx-aks resource_group_name = education-xxxxxx-rg
Now, view the information about the AKS cluster.
$ kubectl cluster-info Kubernetes master is running at https://education.hcp.westus2.azmk8s.io:443 Heapster is running at ... ...
NOTE: Copy the Kubernetes master address (
https://education.hcp.westus2.azmk8s.io:443
in this example).In the
/vault-agent-k8s-demo/setup-k8s-auth.sh
file, replace Line 48 to point to the AKS cluster address rather thanexport K8S_HOST=$(minikube ip)
. Also, replace Line 54 to point to the correct host address.Example (
setup-k8s-auth.sh
):... # Set K8S_HOST to minikube IP address # export K8S_HOST=$(minikube ip) export K8S_HOST="https://education.hcp.westus2.azmk8s.io:443" # Enable the Kubernetes auth method at the default path ("auth/kubernetes") vault auth enable kubernetes # Tell Vault how to communicate with the Kubernetes (Minikube) cluster # vault write auth/kubernetes/config token_reviewer_jwt="$SA_JWT_TOKEN" kubernetes_host="https://$K8S_HOST:8443" kubernetes_ca_cert="$SA_CA_CRT" vault write auth/kubernetes/config token_reviewer_jwt="$SA_JWT_TOKEN" kubernetes_host="$K8S_HOST" kubernetes_ca_cert="$SA_CA_CRT" ...
Set the working directory to where scripts are located (
/vault-agent-k8s-demo/
).$ cd ..
Create a service account,
vault-auth
.$ kubectl create serviceaccount vault-auth
Update the
vault-auth
service account based on the definition in thevault-auth-service-account.yaml
file.$ kubectl apply --filename vault-auth-service-account.yaml
Setup the Kubernetes auth method on the Vault server.
$ ./setup-k8s-auth.sh
Resume Step 3 and on.
»AKS Kubernetes Dashboard
Click on the Monitor containers in the Azure portal.
Click the Enable button so that you can monitor the Kubernetes cluster utilization metrics.
»Clean up
When you are done experimenting this demo, execute the following commands to clean up.
Execute the terraform destroy
command.
$ terraform destroy -force
Remove the Terraform state files.
$ rm -rf .terraform terraform.tfstate*
»Google Kubernetes Engine Cluster
Set your working directory to where the
/identity/vault-agent-k8s-demo/terraform-gcp
folder is located.Modify
terraform.tfvars.example
and provide GCP credentials:account_file_path
andproject
.
Example:
account_file_path = "/usr/student/gcp/vault-test-project.json"
project = "vault-test-project"
Execute the Terraform commands to provision a new GKE cluster.
Pull necessary plugins.
$ terraform init
Now, execute the
apply
command to build a new GKE cluster.$ terraform apply -auto-approve
Connect to the GKE cluster.
$ gcloud container clusters get-credentials $(terraform output gcp_cluster_name) \ --zone $(terraform output gcp_zone) \ --project $(terraform output gcp_project)
Get the cluster information.
$ kubectl cluster-info Kubernetes master is running at https://198.51.100.24 GLBCDefaultBackend is running at https://198.51.100.24/api/v1/namespaces/... ...
NOTE: If you don't have the
gcloud
command line tool installed, follow the online documentation.Copy the Kubernetes master address (
https://198.51.100.24
in this example).In the
/vault-agent-k8s-demo/setup-k8s-auth.sh
file, replace Line 48 to point to the GKE cluster address rather thanexport K8S_HOST=$(minikube ip)
. Also, replace Line 54 to point to the correct host address.Example (
setup-k8s-auth.sh
):... # Set K8S_HOST to minikube IP address # export K8S_HOST=$(minikube ip) export K8S_HOST="https://198.51.100.24" # Enable the Kubernetes auth method at the default path ("auth/kubernetes") vault auth enable kubernetes # Tell Vault how to communicate with the Kubernetes (Minikube) cluster # vault write auth/kubernetes/config token_reviewer_jwt="$SA_JWT_TOKEN" kubernetes_host="https://$K8S_HOST:8443" kubernetes_ca_cert="$SA_CA_CRT" vault write auth/kubernetes/config token_reviewer_jwt="$SA_JWT_TOKEN" kubernetes_host="$K8S_HOST" kubernetes_ca_cert="$SA_CA_CRT" ...
Set the working directory to where scripts are located (
/vault-agent-k8s-demo/
).$ cd ..
Create a service account,
vault-auth
.$ kubectl create serviceaccount vault-auth
Update the
vault-auth
service account with definition provided in thevault-auth-service-account.yaml
file.$ kubectl apply --filename vault-auth-service-account.yaml
Setup the Kubernetes auth method on the Vault server.
$ ./setup-k8s-auth.sh
Resume Step 3 and on.
»Help and Reference
NOTE: To learn about Vault Agent Caching, refer to the Vault Agent Caching tutorial.