Vault Agent with Kubernetes | Vault

Nearly all requests to Vault must be accompanied by an authentication token. This includes all API requests, as well as via the Vault CLI and other libraries. If you can securely get the first secret from an originator to a consumer, all subsequent secrets transmitted between this originator and consumer can be authenticated with the trust established by the successful distribution and user of that first secret.

Secure Introduction

»Challenge

The applications running in a Kubernetes environment is no exception. Luckily, Vault provides Kubernetes auth method to authenticate the clients using a Kubernetes Service Account Token.

Kubernetes

However, the client is still responsible for managing the lifecycle of its Vault tokens. Therefore, the next challenge becomes how to manage the lifecycle of tokens in a standard way without having to write custom logic.

»Solution

Vault Agent provides a number of different helper features, specifically addressing the following challenges:

Automatic authentication
Secure delivery/storage of tokens
Lifecycle management of these tokens (renewal & re-authentication)

Vault Agent

This tutorial focuses on the demonstration of the Vault Agent Auto-Auth using the kubernetes auth method. Vault Helm introduced Agent Sidecar Injector. Refer to Injecting Secrets into Kubernetes Pods via Vault Helm Sidecar for a step-by-step tutorial on the sidecar usage.

»Prerequisites

To perform the tasks described in this tutorial, you need:

Minikube installed
A running Vault environment reachable from your Kubernetes environment. Refer to the Getting Started tutorial to install Vault. Make sure that your Vault server has been initialized and unsealed

NOTE: For the purpose of demonstration, this tutorial runs Minikube as a Kubernetes environment. If you wish to test against an Azure Kubernetes Service (AKS) cluster instead, follow the steps in the Azure Kubernetes Service Cluster section to create an AKS cluster. If you want to test against a Google Kubernetes Engine (GKE) cluster, refer to the steps described in the Google Kubernetes Engine Cluster section.

»Download demo assets

Clone or download the demo assets from the hashicorp/vault-guides GitHub repository to perform the steps described in this tutorial.

»Start a Vault server

To go through this tutorial, start a Vault dev server which listens for requests locally at 0.0.0.0:8200 with root as the root token ID.

$ vault server -dev -dev-root-token-id root -dev-listen-address 0.0.0.0:8200

Setting the -dev-listen-address to 0.0.0.0:8200 overrides the default address of a Vault dev server (127.0.0.1:8200) and enables Vault to be addressable by the Kubernetes cluster and its pods because it binds to a shared network.

Insecure operation: Do not run a Vault dev server in production. This approach is only used here to simplify the unsealing process for this demonstration.

Export an environment variable for the vault CLI to address the Vault server.

$ export VAULT_ADDR=http://0.0.0.0:8200

»Step 1: Create a service account

Start a Kubernetes cluster running in Minikube.
```
$ minikube start
```
Wait a couple of minutes for the minikube environment to become fully available.
```
$ minikube status

host: Running
kubelet: Running
apiserver: Running
kubeconfig: Configured
```

From a command terminal, set your working directory to where the /identity/vault-agent-k8s-demo folder is located. The working directory should contain 2 directories and 14 files.

$ cd vault-guides/identity/vault-agent-k8s-demo

$ tree
.
├── README.md
├── configmap.yaml
├── example-k8s-spec.yaml
├── setup-k8s-auth.sh
├── terraform-azure
│   ├── README.md
│   ├── aks-cluster.tf
│   ├── terraform.tfvars.example
│   └── versions.tf
├── terraform-gcp
│   ├── README.md
│   ├── main.tf
│   ├── terraform.tfvars.example
│   ├── variables.tf
│   └── versions.tf
└── vault-auth-service-account.yaml

2 directories, 14 files

In Kubernetes, a service account provides an identity for processes that run in a Pod so that the processes can contact the API server. Examine the provided vault-auth-service-account.yaml file for the service account definition to be used for this tutorial.

$ cat vault-auth-service-account.yaml
  ---
  apiVersion: rbac.authorization.k8s.io/v1beta1
  kind: ClusterRoleBinding
  metadata:
    name: role-tokenreview-binding
    namespace: default
  roleRef:
    apiGroup: rbac.authorization.k8s.io
    kind: ClusterRole
    name: system:auth-delegator
  subjects:
  - kind: ServiceAccount
    name: vault-auth
    namespace: default

Create a Kubernetes service account named vault-auth.
```
$ kubectl create serviceaccount vault-auth
```

Update the vault-auth service account.

$ kubectl apply --filename vault-auth-service-account.yaml

»Step 2: Configure Kubernetes auth method

Create a read-only policy, myapp-kv-ro in Vault.

$ vault policy write myapp-kv-ro - <<EOF
path "secret/data/myapp/*" {
    capabilities = ["read", "list"]
}
EOF

Create some test data at the secret/myapp path.

$ vault kv put secret/myapp/config username='appuser' \
        password='suP3rsec(et!' \
        ttl='30s'

Set the environment variables to point to the running Minikube environment.
Set the VAULT_SA_NAME environment variable value to the service account you created earlier.
```
$ export VAULT_SA_NAME=$(kubectl get sa vault-auth \
    -o jsonpath="{.secrets[*]['name']}")
```
Set the SA_JWT_TOKEN environment variable value to the service account JWT used to access the TokenReview API
```
$ export SA_JWT_TOKEN=$(kubectl get secret $VAULT_SA_NAME \
    -o jsonpath="{.data.token}" | base64 --decode; echo)
```
Set the SA_CA_CRT environment variable value to the PEM encoded CA cert used to talk to Kubernetes API.
```
$ export SA_CA_CRT=$(kubectl get secret $VAULT_SA_NAME \
    -o jsonpath="{.data['ca\.crt']}" | base64 --decode; echo)
```
Set the K8S_HOST environment variable value to minikube IP address.
```
$ export K8S_HOST=$(minikube ip)
```

Now, enable and configure the Kubernetes auth method.

Enable the Kubernetes auth method at the default path ("auth/kubernetes").

$ vault auth enable kubernetes

Tell Vault how to communicate with the Kubernetes (Minikube) cluster.

$ vault write auth/kubernetes/config \
        token_reviewer_jwt="$SA_JWT_TOKEN" \
        kubernetes_host="https://$K8S_HOST:8443" \
        kubernetes_ca_cert="$SA_CA_CRT"

Create a role named, "example" to map Kubernetes Service Account to Vault policies and default token TTL.

$ vault write auth/kubernetes/role/example \
        bound_service_account_names=vault-auth \
        bound_service_account_namespaces=default \
        policies=myapp-kv-ro \
        ttl=24h

NOTE: The pattern Vault uses to authenticate Pods depends on sharing the JWT token over the network. Given the security model of Vault, this is allowable because Vault is part of the trusted compute base. In general, Kubernetes applications should not share this JWT with other applications, as it allows API calls to be made on behalf of the Pod and can result in unintended access being granted to 3rd parties.

»Step 3: Determine the Vault address

A service bound to all networks on the host, as you configured Vault, is addressable by pods within Minikube's cluster by sending requests to the gateway address of the Kubernetes cluster.

Start a minikube SSH session.

$ minikube ssh
## ... minikube ssh login

Within this SSH session, retrieve the value of the Minikube host.

$ route -n | grep ^0.0.0.0 | awk '{ print $2 }'
192.168.64.1

In this example, the Vault address is http://192.168.64.1:8200.

Verify the connectivity to the Vault server using the address you discovered.

$ curl -s http://192.168.64.1:8200/v1/sys/seal-status | jq
{
  "type": "shamir",
  "initialized": true,
  "sealed": false,
  "t": 1,
  "n": 1,
  "progress": 0,
  "nonce": "",
  "version": "1.4.1+ent",
  "migration": false,
  "cluster_name": "vault-cluster-3de6c2d3",
  "cluster_id": "10fd177e-d55a-d740-0c54-26268ed86e31",
  "recovery_seal": false,
  "storage_type": "inmem"
}

The output displays that Vault is initialized and unsealed. This confirms that pods within your cluster are able to reach Vault given that each pod is configured to use the gateway address.

Exit the Minikube SSH session.
```
$ exit
```
Finally, create a variable named EXTERNAL_VAULT_ADDR to capture the Minikube gateway address.
Example:
```
$ EXTERNAL_VAULT_ADDR="http://192.168.64.1:8200"
```

»Optional: Verify the Kubernetes auth method configuration

Create a Pod with a container running alpine:3.7 image.

$ kubectl run --generator=run-pod/v1 tmp --rm -i --tty \
      --serviceaccount=vault-auth --image alpine:3.7

Once you are inside the container, install cURL and jq tools.
```
/# apk update
/# apk add curl jq
```

Set KUBE_TOKEN to the service account token value.

/# KUBE_TOKEN=$(cat /var/run/secrets/kubernetes.io/serviceaccount/token)
/# echo $KUBE_TOKEN

Authenticate with Vault using the Vault address you discovered earlier and set to EXTERNAL_VAULT_ADDR. In the following example, the value is http://192.168.64.1:8200.

/# curl --request POST \
        --data '{"jwt": "'"$KUBE_TOKEN"'", "role": "example"}' \
        http://192.168.64.1:8200/v1/auth/kubernetes/login | jq
{
  ...
  "auth": {
    "client_token": "s.7cH83AFIdmXXYKsPsSbeESpp",
    "accessor": "8bmYWFW5HtwDHLAoxSiuMZRh",
    "policies": [
      "default",
      "myapp-kv-ro"
    ],
    "token_policies": [
      "default",
      "myapp-kv-ro"
    ],
    "metadata": {
      "role": "example",
      "service_account_name": "vault-auth",
      "service_account_namespace": "default",
      "service_account_secret_name": "vault-auth-token-vqqlp",
      "service_account_uid": "adaca842-f2a7-11e8-831e-080027b85b6a"
    },
    "lease_duration": 86400,
    "renewable": true,
    "entity_id": "2c4624f1-29d6-972a-fb27-729b50dd05e2",
    "token_type": "service"
  }
}

Notice that client_token is successfully generated and myapp-kv-ro policy is attached with the token. The metadata displays that its service account name (service_account_name) is vault-auth.

Lastly, exit the pod.
```
/# exit
```

»Step 4: Leverage Vault Agent Auto-Auth

Now that you have verified that the Kubernetes auth method has been configured on the Vault server, it is time to spin up a client Pod which leverages Vault Agent to automatically authenticate with Vault and retrieve a client token.

First, review the provided configmap.yaml file.

$ cat configmap.yaml

apiVersion: v1
data:
  vault-agent-config.hcl: |
    # Comment this out if running as sidecar instead of initContainer
    exit_after_auth = true

    pid_file = "/home/vault/pidfile"

    auto_auth {
        method "kubernetes" {
            mount_path = "auth/kubernetes"
            config = {
                role = "example"
            }
        }

        sink "file" {
            config = {
                path = "/home/vault/.vault-token"
            }
        }
    }

    template {
    destination = "/etc/secrets/index.html"
    contents = <<EOT
    <html>
    <body>
    <p>Some secrets:</p>
    {{- with secret "secret/data/myapp/config" }}
    <ul>
    <li><pre>username: {{ .Data.data.username }}</pre></li>
    <li><pre>password: {{ .Data.data.password }}</pre></li>
    </ul>
    {{ end }}
    </body>
    </html>
    EOT
    }
kind: ConfigMap
metadata:
  name: example-vault-agent-config
  namespace: default

This creates a Vault Agent configuration file, vault-agent-config.hcl. Notice that the Vault Agent Auto-Auth (auto_auth block) is configured to use the kubernetes auth method enabled at the auth/kubernetes path on the Vault server. The Vault Agent will use the example role which you created in Step 2.

The sink block specifies the location on disk where to write tokens. Vault Agent Auto-Auth sink can be configured multiple times if you want Vault Agent to place the token into multiple locations. In this example, the sink is set to /home/vault/.vault-token.

Finally, the template block creates a templated file which retrieves username and password values at the secret/data/myapp/config path.

Create a ConfigMap containing a Vault Agent configuration.
```
$ kubectl create -f configmap.yaml
```

View the created ConfigMap.

$ kubectl get configmap example-vault-agent-config -o yaml

An example Pod spec file is provided. Review the provided example Pod spec file, example-k8s-spec.yaml.

$ cat example-k8s-spec.yaml

apiVersion: v1
kind: Pod
metadata:
  name: vault-agent-example
  namespace: default
spec:
  serviceAccountName: vault-auth

  volumes:
  - configMap:
      items:
      - key: vault-agent-config.hcl
        path: vault-agent-config.hcl
      name: example-vault-agent-config
    name: config
  - emptyDir: {}
    name: shared-data

  initContainers:
  - args:
    - agent
    - -config=/etc/vault/vault-agent-config.hcl
    - -log-level=debug
    env:
    - name: VAULT_ADDR
      value: http://192.168.64.1:8200
    image: vault
    name: vault-agent
    volumeMounts:
    - mountPath: /etc/vault
      name: config
    - mountPath: /etc/secrets
      name: shared-data

  containers:
  - image: nginx
    name: nginx-container
    ports:
    - containerPort: 80
    volumeMounts:
    - mountPath: /usr/share/nginx/html
      name: shared-data

The example Pod spec (example-k8s-spec.yaml) spins up two containers in vault-agent-example Pod. A vault container which runs Vault Agent as an Init Container. And an nginx container exposing port 80.

Kubernetes

NOTE: Set the VAULT_ADDR value to where your Vault server is running. If you are following the Minikube steps, this value should be the EXTERNAL_VAULT_ADDR value you discovered in Step 3.

Create the vault-agent-example Pod defined in example-k8s-spec.yaml.
```
$ kubectl apply -f example-k8s-spec.yaml --record
```
This takes a minute or so for the Pod to become fully up and running.

»Verification

In another terminal, launch the Minikube dashboard.
```
$ minikube dashboard
```
Click Pods under Workloads to verify that vault-agent-example Pod has been created successfully.
1. Select vault-agent-example to see its details.
Port-forward so you can connect to the client from browser.
```
$ kubectl port-forward pod/vault-agent-example 8080:80
```
In a web browser, go to localhost:8080
Notice that the username and password values were successfully read from secret/myapp/config.

Optionally, you can view the HTML source.

$ kubectl exec -it vault-agent-example --container nginx-container sh

/# cat /usr/share/nginx/html/index.html
  <html>
  <body>
  <p>Some secrets:</p>
  <ul>
  <li><pre>username: appuser</pre></li>
  <li><pre>password: suP3rsec(et!</pre></li>
  </ul>

  </body>
  </html>

»Azure Kubernetes Service Cluster

If you wish to test the Kubernetes auth method against an Azure Kubernetes Service (AKS) cluster instead of Minikube, you can run Terraform to provision an AKS cluster.

First, follow the instruction in the Terraform documentation to create a service principal.

Be sure to add the following Azure Active Directory Graph API permission to your app.

API permissions

Store the credentials as Environment Variables.

$ export ARM_CLIENT_ID="00000000-0000-0000-0000-000000000000"
$ export ARM_CLIENT_SECRET="00000000-0000-0000-0000-000000000000"
$ export ARM_SUBSCRIPTION_ID="00000000-0000-0000-0000-000000000000"
$ export ARM_TENANT_ID="00000000-0000-0000-0000-000000000000"

Subscription ID: Navigate to the Subscriptions blade within the Azure Portal and copy the SUBSCRIPTION ID
Tenant ID: Navigate to the Azure Active Directory > Properties in the Azure Portal, and copy the Directory ID which is your tenant ID
Client ID: Same as the Application ID
Client secret: The password (credential) set on your application

Set your working directory to where the /identity/vault-agent-k8s-demo/terraform-azure folder is located.
Modify terraform.tfvars.example and provide Azure credentials: client_id and client_secret, and save it as terraform.tfvars.

Client ID: Same as the Application ID
Client secret: The password (credential) set on your application

Execute the Terraform commands to provision a new AKS cluster.
Pull necessary plugins.
```
$ terraform init
```
Now, execute the apply command to build a new AKS cluster.
```
$ terraform apply -auto-approve
```
NOTE: If you received Resource group 'education-XXXXX-rg' could not be found error, run terraform refresh command.
```
$ terraform refresh
...

Outputs:

kubernetes_cluster_name = education-xxxxxx-aks
resource_group_name = education-xxxxxx-rg
```
Now, view the information about the AKS cluster.
```
$ kubectl cluster-info

Kubernetes master is running at https://education.hcp.westus2.azmk8s.io:443
Heapster is running at ...
...
```
NOTE: Copy the Kubernetes master address (https://education.hcp.westus2.azmk8s.io:443 in this example).

In the /vault-agent-k8s-demo/setup-k8s-auth.sh file, replace Line 48 to point to the AKS cluster address rather than export K8S_HOST=$(minikube ip). Also, replace Line 54 to point to the correct host address.

Example (setup-k8s-auth.sh):

...
# Set K8S_HOST to minikube IP address
# export K8S_HOST=$(minikube ip)
export K8S_HOST="https://education.hcp.westus2.azmk8s.io:443"

# Enable the Kubernetes auth method at the default path ("auth/kubernetes")
vault auth enable kubernetes

# Tell Vault how to communicate with the Kubernetes (Minikube) cluster
# vault write auth/kubernetes/config token_reviewer_jwt="$SA_JWT_TOKEN" kubernetes_host="https://$K8S_HOST:8443" kubernetes_ca_cert="$SA_CA_CRT"
vault write auth/kubernetes/config token_reviewer_jwt="$SA_JWT_TOKEN" kubernetes_host="$K8S_HOST" kubernetes_ca_cert="$SA_CA_CRT"
...

Set the working directory to where scripts are located (/vault-agent-k8s-demo/).
```
$ cd ..
```

Create a service account, vault-auth.

$ kubectl create serviceaccount vault-auth

Update the vault-auth service account based on the definition in the vault-auth-service-account.yaml file.
```
$ kubectl apply --filename vault-auth-service-account.yaml
```
Setup the Kubernetes auth method on the Vault server.
```
$ ./setup-k8s-auth.sh
```
Resume Step 3 and on.

»AKS Kubernetes Dashboard

Click on the Monitor containers in the Azure portal.

Kubernetes

Click the Enable button so that you can monitor the Kubernetes cluster utilization metrics.

»Clean up

When you are done experimenting this demo, execute the following commands to clean up.

Execute the terraform destroy command.

$ terraform destroy -force

Remove the Terraform state files.

$ rm -rf .terraform terraform.tfstate*

»Google Kubernetes Engine Cluster

Set your working directory to where the /identity/vault-agent-k8s-demo/terraform-gcp folder is located.
Modify terraform.tfvars.example and provide GCP credentials: account_file_path and project.

Example:

account_file_path = "/usr/student/gcp/vault-test-project.json"
project  = "vault-test-project"

Execute the Terraform commands to provision a new GKE cluster.
Pull necessary plugins.
```
$ terraform init
```
Now, execute the apply command to build a new GKE cluster.
```
$ terraform apply -auto-approve
```

Connect to the GKE cluster.

$ gcloud container clusters get-credentials $(terraform output gcp_cluster_name) \
        --zone $(terraform output gcp_zone) \
        --project $(terraform output gcp_project)

Get the cluster information.

$ kubectl cluster-info

Kubernetes master is running at https://198.51.100.24
GLBCDefaultBackend is running at https://198.51.100.24/api/v1/namespaces/...
...

NOTE: If you don't have the gcloud command line tool installed, follow the online documentation.

Copy the Kubernetes master address (https://198.51.100.24 in this example).

In the /vault-agent-k8s-demo/setup-k8s-auth.sh file, replace Line 48 to point to the GKE cluster address rather than export K8S_HOST=$(minikube ip). Also, replace Line 54 to point to the correct host address.

Example (setup-k8s-auth.sh):

...
# Set K8S_HOST to minikube IP address
# export K8S_HOST=$(minikube ip)
export K8S_HOST="https://198.51.100.24"

# Enable the Kubernetes auth method at the default path ("auth/kubernetes")
vault auth enable kubernetes

# Tell Vault how to communicate with the Kubernetes (Minikube) cluster
# vault write auth/kubernetes/config token_reviewer_jwt="$SA_JWT_TOKEN" kubernetes_host="https://$K8S_HOST:8443" kubernetes_ca_cert="$SA_CA_CRT"
vault write auth/kubernetes/config token_reviewer_jwt="$SA_JWT_TOKEN" kubernetes_host="$K8S_HOST" kubernetes_ca_cert="$SA_CA_CRT"
...

Set the working directory to where scripts are located (/vault-agent-k8s-demo/).
```
$ cd ..
```

Create a service account, vault-auth.

$ kubectl create serviceaccount vault-auth

Update the vault-auth service account with definition provided in the vault-auth-service-account.yaml file.
```
$ kubectl apply --filename vault-auth-service-account.yaml
```
Setup the Kubernetes auth method on the Vault server.
```
$ ./setup-k8s-auth.sh
```
Resume Step 3 and on.

»Help and Reference

NOTE: To learn about Vault Agent Caching, refer to the Vault Agent Caching tutorial.

Infrastructure

Security

Networking

Applications

»Challenge

»Solution

»Prerequisites

»Download demo assets

»Start a Vault server

»Step 1: Create a service account

»Step 2: Configure Kubernetes auth method

»Step 3: Determine the Vault address

»Optional: Verify the Kubernetes auth method configuration

»Step 4: Leverage Vault Agent Auto-Auth

»Verification

»Azure Kubernetes Service Cluster

»AKS Kubernetes Dashboard

»Clean up

»Google Kubernetes Engine Cluster

»Help and Reference