Access Management

Vault Agent with Kubernetes

Nearly all requests to Vault must be accompanied by an authentication token. This includes all API requests, as well as via the Vault CLI and other libraries. If you can securely get the first secret from an originator to a consumer, all subsequent secrets transmitted between this originator and consumer can be authenticated with the trust established by the successful distribution and user of that first secret.

Secure Introduction

Challenge

The applications running in a Kubernetes environment is no exception. Luckily, Vault provides Kubernetes auth method to authenticate the clients using a Kubernetes Service Account Token.

Kubernetes

However, the client is still responsible for managing the lifecycle of its Vault tokens. Therefore, the next challenge becomes how to manage the lifecycle of tokens in a standard way without having to write custom logic.

Solution

Vault Agent provides a number of different helper features, specifically addressing the following challenges:

  • Automatic authentication
  • Secure delivery/storage of tokens
  • Lifecycle management of these tokens (renewal & re-authentication)

Vault Agent

Prerequisites

To perform the tasks described in this guide, you need:

Download demo assets

Clone or download the demo assets from the hashicorp/vault-guides GitHub repository to perform the steps described in this guide.

Step 1: Create a service account

Execute the following command to start Minikube if it hasn't been started:

$ minikube start

Wait for a couple of minutes for the minikube environment to become fully available.

Open a command terminal, and set your working directory to where the /identity/vault-agent-k8s-demo folder is located. The working directory should contain the following:

$ cd vault-guides/identity/vault-agent-k8s-demo

$ tree
.
├── README.md
├── configs-k8s
│   ├── consul-template-config.hcl
│   └── vault-agent-config.hcl
├── example-k8s-spec.yml
├── setup-k8s-auth.sh
├── terraform-azure
│   ├── README.md
│   ├── aks-cluster.tf
│   ├── terraform.tfvars.example
│   └── versions.tf
├── terraform-gcp
│   ├── README.md
│   ├── main.tf
│   ├── terraform.tfvars.example
│   ├── variables.tf
│   └── versions.tf
└── vault-auth-service-account.yml

3 directories, 15 files

In Kubernetes, a service account provides an identity for processes that run in a Pod so that the processes can contact the API server.

See the provided vault-auth-service-account.yml file for the service account definition to be used for this guide:

    $ cat vault-auth-service-account.yml
      ---
      apiVersion: rbac.authorization.k8s.io/v1beta1
      kind: ClusterRoleBinding
      metadata:
        name: role-tokenreview-binding
        namespace: default
      roleRef:
        apiGroup: rbac.authorization.k8s.io
        kind: ClusterRole
        name: system:auth-delegator
      subjects:
      - kind: ServiceAccount
        name: vault-auth
        namespace: default

Now, let's create a Kubernetes service account named vault-auth.

# Create a service account, 'vault-auth'
$ kubectl create serviceaccount vault-auth

# Update the 'vault-auth' service account
$ kubectl apply --filename vault-auth-service-account.yml

Step 2: Configure Kubernetes auth method

  1. Create a read-only policy, myapp-kv-ro in Vault.

    # Create a policy file, myapp-kv-ro.hcl
    $ tee myapp-kv-ro.hcl <<EOF
    # If working with K/V v1
    path "secret/myapp/*" {
        capabilities = ["read", "list"]
    }
    
    # If working with K/V v2
    path "secret/data/myapp/*" {
        capabilities = ["read", "list"]
    }
    EOF
    
    # Create a policy named myapp-kv-ro
    $ vault policy write myapp-kv-ro myapp-kv-ro.hcl
    
  2. Create some secrets at the secret/myapp path for testing.

    $ vault kv put secret/myapp/config username='appuser' \
            password='suP3rsec(et!' \
            ttl='30s'
    
  3. Create a user to test the myapp-kv-ro policy using userpass auth method.

    # Enable userpass auth method
    $ vault auth enable userpass
    
    # Create a user named "test-user"
    $ vault write auth/userpass/users/test-user \
            password=training \
            policies=myapp-kv-ro
    
    # Login as test-user
    $ vault login -method=userpass \
            username=test-user \
            password=training
    
    # Test to see if test-user can read secret/myapp path as policy has written
    $ vault kv get secret/myapp/config
    
  4. Now, log back in with privileged token so that you can enable and configure an auth method.

  5. Set the environment variables to point to the running Minikube environment.

    # Set VAULT_SA_NAME to the service account you created earlier
    $ export VAULT_SA_NAME=$(kubectl get sa vault-auth -o jsonpath="{.secrets[*]['name']}")
    
    # Set SA_JWT_TOKEN value to the service account JWT used to access the TokenReview API
    $ export SA_JWT_TOKEN=$(kubectl get secret $VAULT_SA_NAME -o jsonpath="{.data.token}" | base64 --decode; echo)
    
    # Set SA_CA_CRT to the PEM encoded CA cert used to talk to Kubernetes API
    $ export SA_CA_CRT=$(kubectl get secret $VAULT_SA_NAME -o jsonpath="{.data['ca\.crt']}" | base64 --decode; echo)
    
    # Set K8S_HOST to minikube IP address
    $ export K8S_HOST=$(minikube ip)
    
  6. Now, enable and configure the Kubernetes auth method.

    # Enable the Kubernetes auth method at the default path ("auth/kubernetes")
    $ vault auth enable kubernetes
    
    # Tell Vault how to communicate with the Kubernetes (Minikube) cluster
    $ vault write auth/kubernetes/config \
            token_reviewer_jwt="$SA_JWT_TOKEN" \
            kubernetes_host="https://$K8S_HOST:8443" \
            kubernetes_ca_cert="$SA_CA_CRT"
    
    # Create a role named, 'example' to map Kubernetes Service Account to
    #  Vault policies and default token TTL
    $ vault write auth/kubernetes/role/example \
            bound_service_account_names=vault-auth \
            bound_service_account_namespaces=default \
            policies=myapp-kv-ro \
            ttl=24h
    

Step 3: Verify the Kubernetes auth method configuration

Create a Pod with a container running alpine:3.7 image.

$ kubectl run --generator=run-pod/v1 tmp --rm -i --tty --serviceaccount=vault-auth --image alpine:3.7

Once you are inside the container, install cURL and jq tools.

/# apk update
/# apk add curl jq

Set the VAULT_ADDR environment variable to point to the running Vault where you configured Kubernetes auth method, and test the connection.

/# VAULT_ADDR=http://10.0.2.2:8200

/# curl -s $VAULT_ADDR/v1/sys/health | jq
{
  "initialized": true,
  "sealed": false,
  "standby": false,
  "performance_standby": false,
  "replication_performance_mode": "disabled",
  "replication_dr_mode": "disabled",
  "server_time_utc": 1543969628,
  "version": "1.0.0+ent",
  "cluster_name": "vault-cluster-e314942e",
  "cluster_id": "2b4f6213-d58f-0530-cf07-65ea467181f2"
}

Set KUBE_TOKEN to the service account token value:

/# KUBE_TOKEN=$(cat /var/run/secrets/kubernetes.io/serviceaccount/token)
/# echo $KUBE_TOKEN

Now, test the kubernetes auth method to ensure that you can authenticate with Vault.

/# curl --request POST \
        --data '{"jwt": "'"$KUBE_TOKEN"'", "role": "example"}' \
        $VAULT_ADDR/v1/auth/kubernetes/login | jq
{
  ...
  "auth": {
    "client_token": "s.7cH83AFIdmXXYKsPsSbeESpp",
    "accessor": "8bmYWFW5HtwDHLAoxSiuMZRh",
    "policies": [
      "default",
      "myapp-kv-ro"
    ],
    "token_policies": [
      "default",
      "myapp-kv-ro"
    ],
    "metadata": {
      "role": "example",
      "service_account_name": "vault-auth",
      "service_account_namespace": "default",
      "service_account_secret_name": "vault-auth-token-vqqlp",
      "service_account_uid": "adaca842-f2a7-11e8-831e-080027b85b6a"
    },
    "lease_duration": 86400,
    "renewable": true,
    "entity_id": "2c4624f1-29d6-972a-fb27-729b50dd05e2",
    "token_type": "service"
  }
}

Notice that client_token is successfully generated and myapp-kv-ro policy is attached with the token. The metadata displays that its service account name (service_account_name) is vault-auth.

Enter exit to terminate the shell session.

Step 4: Leverage Vault Agent Auto-Auth

Now that you have verified that the Kubernetes auth method has been configured on the Vault server, it is time to spin up a client Pod which leverages Vault Agent to automatically authenticate with Vault and retrieve a client token.

Kubernetes

Review the provided Vault Agent configuration file, vault-agent-config.hcl which is located in the vault-guides/identity/vault-agent-k8s-demo/configs-k8s folder.

exit_after_auth = true
pid_file = "/home/vault/pidfile"

auto_auth {
    method "kubernetes" {
        mount_path = "auth/kubernetes"
        config = {
            role = "example"
        }
    }

    sink "file" {
        config = {
            path = "/home/vault/.vault-token"
        }
    }
}

Notice that the Vault Agent Auto-Auth is configured to use the kubernetes auth method enabled at the auth/kubernetes path on the Vault server. The Vault Agent will use the example role to authenticate.

The sink block specifies the location on disk where to write tokens. Vault Agent Auto-Auth sink can be configured multiple times if you want Vault Agent to place the token into multiple locations. In this example, the sink is set to /home/vault/.vault-token.

Now, review the Consul Template file, consul-template-config.hcl which is located in the vault-guides/identity/vault-agent-k8s-demo/configs-k8s folder.

vault {
  renew_token = false
  vault_agent_token_file = "/home/vault/.vault-token"
  retry {
    backoff = "1s"
  }
}

template {
  destination = "/etc/secrets/index.html"
  contents = <<EOH
  <html>
  <body>
  <p>Some secrets:</p>
  {{- with secret "secret/myapp/config" }}
  <ul>
  <li><pre>username: {{ .Data.username }}</pre></li>
  <li><pre>password: {{ .Data.password }}</pre></li>
  </ul>
  {{ end }}
  </body>
  </html>
  EOH
}

This template reads secrets at the secret/myapp/config path and set the username and password values. (If you are unfamiliar with Consul Template, refer to Direct Application Integration guide.)

NOTE: If the secret/ path is enabled with key/value v2 secrets engine, the templatized expressions should be modified as follow (Line 15 through 18):

...
template {
  ...
  {{- with secret "secret/data/myapp/config?version=1" }}
  <ul>
  <li><pre>username: {{ .Data.data.username }}</pre></li>
  <li><pre>password: {{ .Data.data.password }}</pre></li>
  </ul>
  {{ end }}
  ...
}

In Kubernetes, ConfigMaps allow you to decouple configuration artifacts from image content to keep containerized applications portable. Now, create a ConfigMap named, example-vault-agent-config pulling files from configs-k8s directory.

# Create a ConfigMap, example-vault-agent-config
$ kubectl create configmap example-vault-agent-config --from-file=./configs-k8s/

# View the created ConfigMap
$ kubectl get configmap example-vault-agent-config -o yaml

An example Pod spec file is provided. Review the provided example Pod spec file (/vault-agent-k8s-demo/example-k8s-spec.yml):

$ cat example-k8s-spec.yml

---
apiVersion: v1
kind: Pod
metadata:
  name: vault-agent-example
spec:
  serviceAccountName: vault-auth

  ...
  initContainers:
    # Vault container
    - name: vault-agent-auth
      image: vault

      volumeMounts:
        - name: config
          mountPath: /etc/vault
        - name: vault-token
          mountPath: /home/vault

      # This assumes Vault running on local host and K8s running in Minikube using VirtualBox
      env:
        - name: VAULT_ADDR
          value: http://10.0.2.2:8200
      ...
  containers:
    # Consul Template container
    - name: consul-template
      image: hashicorp/consul-template:alpine
      imagePullPolicy: Always
      ...
      env:
        - name: HOME
          value: /home/vault

        - name: VAULT_ADDR
          value: http://10.0.2.2:8200
       ...

    # Nginx container
    - name: nginx-container
      image: nginx
      ...

The example Pod spec (example-k8s-spec.yml) spins up three containers in vault-agent-example Pod:

  • A vault container (starting at line 31) which runs Vault Agent (starting at line 47). NOTE: This container runs as an Init Container

  • A consul-template container (line 56) which runs the Consul Template with template file, consul-template-config.hcl (starting at line 77)

  • nginx container exposing port 80 (starting at line 85)


Execute the following command to create the vault-agent-example Pod:

$ kubectl apply -f example-k8s-spec.yml --record

This takes a minute or so for the Pod to become fully up and running.

Verification

Open another terminal and launch the Minikube dashboard:

$ minikube dashboard

Click Pods under Workloads to verify that vault-agent-example Pod has been created successfully.

Kubernetes

Select vault-agent-example to see its details.

Kubernetes

Now, port-forward so you can connect to the client from browser:

$ kubectl port-forward pod/vault-agent-example 8080:80

In a web browser, go to localhost:8080

Kubernetes

Notice that the username and password values were successfully read from secret/myapp/config.

Open a shell of consul-template container:

$ kubectl exec -it vault-agent-example --container consul-template sh

Remember that the Vault Agent's sink is set to /home/vault/.vault-token. To view the token stored in the sink:

/# echo $(cat /home/vault/.vault-token)
s.7MQZzFZxUTBQMrtfy98wTGkZ

Enter exit to terminate the shell.

Optionally, you can view the HTML source:

$ kubectl exec -it vault-agent-example --container nginx-container sh

/# cat /usr/share/nginx/html/index.html
  <html>
  <body>
  <p>Some secrets:</p>
  <ul>
  <li><pre>username: appuser</pre></li>
  <li><pre>password: suP3rsec(et!</pre></li>
  </ul>

  </body>
  </html>

Azure Kubernetes Service Cluster

If you wish to test the Kubernetes auth method against an Azure Kubernetes Service (AKS) cluster instead of Minikube, you can run Terraform to provision an AKS cluster.

First, follow the instruction in the Terraform documentation to create a service principal.

Be sure to add the following Azure Active Directory Graph API permission to your app.

API permissions

Store the credentials as Environment Variables:

$ export ARM_CLIENT_ID="00000000-0000-0000-0000-000000000000"
$ export ARM_CLIENT_SECRET="00000000-0000-0000-0000-000000000000"
$ export ARM_SUBSCRIPTION_ID="00000000-0000-0000-0000-000000000000"
$ export ARM_TENANT_ID="00000000-0000-0000-0000-000000000000"
  1. Set your working directory to where the /identity/vault-agent-k8s-demo/terraform-azure folder is located.

  2. Modify terraform.tfvars.example and provide Azure credentials: client_id and client_secret, and save it as terraform.tfvars.

  1. Execute the following Terraform commands:

    # Pull necessary plugins
    $ terraform init
    
    # Create an execution plan
    $ terraform plan
    
    # Apply to create a new AKS cluster
    $ terraform apply -auto-approve
    
    $ terraform refresh
    ...
    
    Outputs:
    
    kubernetes_cluster_name = education-xxxxxx-aks
    resource_group_name = education-xxxxxx-rg
    
  2. Now, you should be able to start working with the AKS cluster:

    $ kubectl cluster-info
    
    Kubernetes master is running at https://education.hcp.westus2.azmk8s.io:443
    Heapster is running at ...
    ...
    

    NOTE: Copy the Kubernetes master address (https://education.hcp.westus2.azmk8s.io:443 in this example).

  3. In the /vault-agent-k8s-demo/setup-k8s-auth.sh file, replace Line 48 to point to the AKS cluster address rather than export K8S_HOST=$(minikube ip). Also, replace Line 54 to point to the correct host address.

    Example (setup-k8s-auth.sh):

    ...
    # Set K8S_HOST to minikube IP address
    # export K8S_HOST=$(minikube ip)
    export K8S_HOST="https://education.hcp.westus2.azmk8s.io:443"
    
    # Enable the Kubernetes auth method at the default path ("auth/kubernetes")
    vault auth enable kubernetes
    
    # Tell Vault how to communicate with the Kubernetes (Minikube) cluster
    # vault write auth/kubernetes/config token_reviewer_jwt="$SA_JWT_TOKEN" kubernetes_host="https://$K8S_HOST:8443" kubernetes_ca_cert="$SA_CA_CRT"
    vault write auth/kubernetes/config token_reviewer_jwt="$SA_JWT_TOKEN" kubernetes_host="$K8S_HOST" kubernetes_ca_cert="$SA_CA_CRT"
    ...
    
  4. Execute the following commands:

    # Set the working directory to where scripts are located (/vault-agent-k8s-demo/)
    $ cd ..
    
    # Create a service account, 'vault-auth'
    $ kubectl create serviceaccount vault-auth
    
    # Update the 'vault-auth' service account
    $ kubectl apply --filename vault-auth-service-account.yml
    
    # Setup Kubernetes auth method
    # You should be logged in to Vault with appropriate permissions
    $ ./setup-k8s-auth.sh
    
  5. Resume Step 3 and on.

AKS Kubernetes Dashboard

Clicking on the Monitor containers in the Azure portal.

Kubernetes

Click the Enable button so that you can monitor the Kubernetes cluster utilization metrics.

Clean up

When you are done experimenting this demo, execute the following commands to clean up:

$ terraform destroy -force

$ rm -rf .terraform terraform.tfstate*

Google Kubernetes Engine Cluster

  1. Set your working directory to where the /identity/vault-agent-k8s-demo/terraform-gcp folder is located.

  2. Modify terraform.tfvars.example and provide GCP credentials: account_file_path and project.

    Example:

    account_file_path = "/usr/student/gcp/vault-test-project.json"
    project  = "vault-test-project"
    
  3. Execute the following Terraform commands:

    # Pull necessary plugins
    $ terraform init
    
    # Create an execution plan
    $ terraform plan
    
    # Apply to create a new GKE cluster
    $ terraform apply -auto-approve
    
  4. Now, you should be able to start working with the GKE cluster:

    # Connect to your GKE cluster
    $ gcloud container clusters get-credentials $(terraform output gcp_cluster_name) \
            --zone $(terraform output gcp_zone) \
            --project $(terraform output gcp_project)
    
    # Now, you should be able to get the cluster info via kubectl
    $ kubectl cluster-info
    
    Kubernetes master is running at https://198.51.100.24
    GLBCDefaultBackend is running at https://198.51.100.24/api/v1/namespaces/...
    ...
    

    NOTE: If you don't have the gcloud command line tool installed, follow the online documentation.

  5. Copy the Kubernetes master address (https://198.51.100.24 in this example).

  6. In the /vault-agent-k8s-demo/setup-k8s-auth.sh file, replace Line 48 to point to the GKE cluster address rather than export K8S_HOST=$(minikube ip). Also, replace Line 54 to point to the correct host address.

    Example (setup-k8s-auth.sh):

    ...
    # Set K8S_HOST to minikube IP address
    # export K8S_HOST=$(minikube ip)
    export K8S_HOST="https://198.51.100.24"
    
    # Enable the Kubernetes auth method at the default path ("auth/kubernetes")
    vault auth enable kubernetes
    
    # Tell Vault how to communicate with the Kubernetes (Minikube) cluster
    # vault write auth/kubernetes/config token_reviewer_jwt="$SA_JWT_TOKEN" kubernetes_host="https://$K8S_HOST:8443" kubernetes_ca_cert="$SA_CA_CRT"
    vault write auth/kubernetes/config token_reviewer_jwt="$SA_JWT_TOKEN" kubernetes_host="$K8S_HOST" kubernetes_ca_cert="$SA_CA_CRT"
    ...
    
  7. Execute the following commands:

    # Set the working directory to where scripts are located (/vault-agent-k8s-demo/)
    $ cd ..
    
    # Create a service account, 'vault-auth'
    $ kubectl create serviceaccount vault-auth
    
    # Update the 'vault-auth' service account
    $ kubectl apply --filename vault-auth-service-account.yml
    
    # Setup Kubernetes auth method
    # You should be logged in to Vault with appropriate permissions
    $ ./setup-k8s-auth.sh
    
  8. Resume Step 3 and on.

Help and Reference