NewHCP Vault is now Generally Available Sign up today

Vault Agent with Kubernetes

Nearly all requests to Vault must be accompanied by an authentication token. This includes all API requests, as well as via the Vault CLI and other libraries. If you can securely get the first secret from an originator to a consumer, all subsequent secrets transmitted between this originator and consumer can be authenticated with the trust established by the successful distribution and user of that first secret.

Secure Introduction

»Challenge

The applications running in a Kubernetes environment is no exception. Luckily, Vault provides Kubernetes auth method to authenticate the clients using a Kubernetes Service Account Token.

Kubernetes

However, the client is still responsible for managing the lifecycle of its Vault tokens. Therefore, the next challenge becomes how to manage the lifecycle of tokens in a standard way without having to write custom logic.

»Solution

Vault Agent provides a number of different helper features, specifically addressing the following challenges:

  • Automatic authentication
  • Secure delivery/storage of tokens
  • Lifecycle management of these tokens (renewal & re-authentication)

Vault Agent

»Prerequisites

To perform the tasks described in this tutorial, you need:

»Start a Vault server

To go through this tutorial, start a Vault dev server which listens for requests locally at 0.0.0.0:8200 with root as the root token ID.

$ vault server -dev -dev-root-token-id root -dev-listen-address 0.0.0.0:8200

Setting the -dev-listen-address to 0.0.0.0:8200 overrides the default address of a Vault dev server (127.0.0.1:8200) and enables Vault to be addressable by the Kubernetes cluster and its pods because it binds to a shared network.

Export an environment variable for the vault CLI to address the Vault server.

$ export VAULT_ADDR=http://0.0.0.0:8200

»Create a service account

  1. Start a Kubernetes cluster running in Minikube.

    $ minikube start

    Wait a couple of minutes for the minikube environment to become fully available.

    $ minikube status

host: Running
kubelet: Running
apiserver: Running
kubeconfig: Configured

  2. Clone the demo assets from the hashicorp/vault-guides GitHub repository to perform the steps described in this tutorial.

    $ git clone https://github.com/hashicorp/vault-guides.git

    This repository contains supporting content for all of the Vault learn tutorials. The content specific to this tutorial can be found within a sub-directory.

  3. From a command terminal, set your working directory to where the /identity/vault-agent-k8s-demo folder is located.

    $ cd vault-guides/identity/vault-agent-k8s-demo

    The working directory should contain 2 directories and 14 files.

    $ tree
.
├── README.md
├── configmap.yaml
├── example-k8s-spec.yaml
├── setup-k8s-auth.sh
├── terraform-azure
│   ├── README.md
│   ├── aks-cluster.tf
│   ├── terraform.tfvars.example
│   └── versions.tf
├── terraform-gcp
│   ├── README.md
│   ├── main.tf
│   ├── terraform.tfvars.example
│   ├── variables.tf
│   └── versions.tf
└── vault-auth-service-account.yaml

2 directories, 14 files

  4. In Kubernetes, a service account provides an identity for processes that run in a Pod so that the processes can contact the API server. Examine the provided vault-auth-service-account.yaml file for the service account definition to be used for this tutorial.

    $ cat vault-auth-service-account.yaml
  ---
  apiVersion: rbac.authorization.k8s.io/v1beta1
  kind: ClusterRoleBinding
  metadata:
    name: role-tokenreview-binding
    namespace: default
  roleRef:
    apiGroup: rbac.authorization.k8s.io
    kind: ClusterRole
    name: system:auth-delegator
  subjects:
  - kind: ServiceAccount
    name: vault-auth
    namespace: default

  5. Create a Kubernetes service account named vault-auth.

    $ kubectl create serviceaccount vault-auth

  6. Update the vault-auth service account.

    $ kubectl apply --filename vault-auth-service-account.yaml

»Configure Kubernetes auth method

  1. Create a read-only policy, myapp-kv-ro in Vault.

    $ vault policy write myapp-kv-ro - <<EOF
path "secret/data/myapp/*" {
    capabilities = ["read", "list"]
}
EOF

  2. Create some test data at the secret/myapp path.

    $ vault kv put secret/myapp/config username='appuser' \
        password='suP3rsec(et!' \
        ttl='30s'

  3. Set the environment variables to point to the running Minikube environment.

    Set the VAULT_SA_NAME environment variable value to the service account you created earlier.

    $ export VAULT_SA_NAME=$(kubectl get sa vault-auth \
    -o jsonpath="{.secrets[*]['name']}")

    Set the SA_JWT_TOKEN environment variable value to the service account JWT used to access the TokenReview API

    $ export SA_JWT_TOKEN=$(kubectl get secret $VAULT_SA_NAME \
    -o jsonpath="{.data.token}" | base64 --decode; echo)

    Set the SA_CA_CRT environment variable value to the PEM encoded CA cert used to talk to Kubernetes API.

    $ export SA_CA_CRT=$(kubectl get secret $VAULT_SA_NAME \
    -o jsonpath="{.data['ca\.crt']}" | base64 --decode; echo)

    Set the K8S_HOST environment variable value to minikube IP address.

    $ export K8S_HOST=$(minikube ip)

  4. Now, enable and configure the Kubernetes auth method.

    Enable the Kubernetes auth method at the default path ("auth/kubernetes").

    $ vault auth enable kubernetes

    Tell Vault how to communicate with the Kubernetes (Minikube) cluster.

    $ vault write auth/kubernetes/config \
        token_reviewer_jwt="$SA_JWT_TOKEN" \
        kubernetes_host="https://$K8S_HOST:8443" \
        kubernetes_ca_cert="$SA_CA_CRT"

  5. Create a role named, "example" to map Kubernetes Service Account to Vault policies and default token TTL.

    $ vault write auth/kubernetes/role/example \
        bound_service_account_names=vault-auth \
        bound_service_account_namespaces=default \
        policies=myapp-kv-ro \
        ttl=24h

»Determine the Vault address

A service bound to all networks on the host, as you configured Vault, is addressable by pods within Minikube's cluster by sending requests to the gateway address of the Kubernetes cluster.

  1. Start a minikube SSH session.

    $ minikube ssh
## ... minikube ssh login

  2. Within this SSH session, retrieve the value of the Minikube host.

    $ route -n | grep ^0.0.0.0 | awk '{ print $2 }'
192.168.64.1

    In this example, the Vault address is http://192.168.64.1:8200.

  3. Verify the connectivity to the Vault server using the address you discovered.

    $ curl -s http://192.168.64.1:8200/v1/sys/seal-status | jq
{
  "type": "shamir",
  "initialized": true,
  "sealed": false,
  "t": 1,
  "n": 1,
  "progress": 0,
  "nonce": "",
  "version": "1.4.1+ent",
  "migration": false,
  "cluster_name": "vault-cluster-3de6c2d3",
  "cluster_id": "10fd177e-d55a-d740-0c54-26268ed86e31",
  "recovery_seal": false,
  "storage_type": "inmem"
}

    The output displays that Vault is initialized and unsealed. This confirms that pods within your cluster are able to reach Vault given that each pod is configured to use the gateway address.

  4. Exit the Minikube SSH session.

    $ exit

  5. Finally, create a variable named EXTERNAL_VAULT_ADDR to capture the Minikube gateway address.

    Example:

    $ EXTERNAL_VAULT_ADDR="http://192.168.64.1:8200"

»Optional: Verify the Kubernetes auth method configuration

  1. Create a Pod with a container running alpine:3.7 image.

    $ kubectl run --generator=run-pod/v1 tmp --rm -i --tty \
      --serviceaccount=vault-auth --image alpine:3.7

  2. Once you are inside the container, install cURL and jq tools.

    /# apk update
/# apk add curl jq

  3. Set KUBE_TOKEN to the service account token value.

    /# KUBE_TOKEN=$(cat /var/run/secrets/kubernetes.io/serviceaccount/token)
/# echo $KUBE_TOKEN

  4. Authenticate with Vault using the Vault address you discovered earlier and set to EXTERNAL_VAULT_ADDR. In the following example, the value is http://192.168.64.1:8200.

    /# curl --request POST \
        --data '{"jwt": "'"$KUBE_TOKEN"'", "role": "example"}' \
        http://192.168.64.1:8200/v1/auth/kubernetes/login | jq
{
  ...
  "auth": {
    "client_token": "s.7cH83AFIdmXXYKsPsSbeESpp",
    "accessor": "8bmYWFW5HtwDHLAoxSiuMZRh",
    "policies": [
      "default",
      "myapp-kv-ro"
    ],
    "token_policies": [
      "default",
      "myapp-kv-ro"
    ],
    "metadata": {
      "role": "example",
      "service_account_name": "vault-auth",
      "service_account_namespace": "default",
      "service_account_secret_name": "vault-auth-token-vqqlp",
      "service_account_uid": "adaca842-f2a7-11e8-831e-080027b85b6a"
    },
    "lease_duration": 86400,
    "renewable": true,
    "entity_id": "2c4624f1-29d6-972a-fb27-729b50dd05e2",
    "token_type": "service"
  }
}

    Notice that client_token is successfully generated and myapp-kv-ro policy is attached with the token. The metadata displays that its service account name (service_account_name) is vault-auth.

  5. Lastly, exit the pod.

    /# exit

»Start Vault Agent with Auto-Auth

Now that you have verified that the Kubernetes auth method has been configured on the Vault server, it is time to spin up a client Pod which leverages Vault Agent to automatically authenticate with Vault and retrieve a client token.

  1. First, review the provided configmap.yaml file.

    $ cat configmap.yaml

apiVersion: v1
data:
  vault-agent-config.hcl: |
    # Comment this out if running as sidecar instead of initContainer
    exit_after_auth = true

    pid_file = "/home/vault/pidfile"

    auto_auth {
        method "kubernetes" {
            mount_path = "auth/kubernetes"
            config = {
                role = "example"
            }
        }

        sink "file" {
            config = {
                path = "/home/vault/.vault-token"
            }
        }
    }

    template {
    destination = "/etc/secrets/index.html"
    contents = <<EOT
    <html>
    <body>
    <p>Some secrets:</p>
    {{- with secret "secret/data/myapp/config" }}
    <ul>
    <li><pre>username: {{ .Data.data.username }}</pre></li>
    <li><pre>password: {{ .Data.data.password }}</pre></li>
    </ul>
    {{ end }}
    </body>
    </html>
    EOT
    }
kind: ConfigMap
metadata:
  name: example-vault-agent-config
  namespace: default

    This creates a Vault Agent configuration file, vault-agent-config.hcl. Notice that the Vault Agent Auto-Auth (auto_auth block) is configured to use the kubernetes auth method enabled at the auth/kubernetes path on the Vault server. The Vault Agent will use the example role which you created in Step 2.

    The sink block specifies the location on disk where to write tokens. Vault Agent Auto-Auth sink can be configured multiple times if you want Vault Agent to place the token into multiple locations. In this example, the sink is set to /home/vault/.vault-token.

    Finally, the template block creates a templated file which retrieves username and password values at the secret/data/myapp/config path.

  2. Create a ConfigMap containing a Vault Agent configuration.

    $ kubectl create -f configmap.yaml

  3. View the created ConfigMap.

    $ kubectl get configmap example-vault-agent-config -o yaml

  4. An example Pod spec file is provided. Review the provided example Pod spec file, example-k8s-spec.yaml.

    $ cat example-k8s-spec.yaml

apiVersion: v1
kind: Pod
metadata:
  name: vault-agent-example
  namespace: default
spec:
  serviceAccountName: vault-auth

  volumes:
  - configMap:
      items:
      - key: vault-agent-config.hcl
        path: vault-agent-config.hcl
      name: example-vault-agent-config
    name: config
  - emptyDir: {}
    name: shared-data

  initContainers:
  - args:
    - agent
    - -config=/etc/vault/vault-agent-config.hcl
    - -log-level=debug
    env:
    - name: VAULT_ADDR
      value: http://192.168.64.1:8200
    image: vault
    name: vault-agent
    volumeMounts:
    - mountPath: /etc/vault
      name: config
    - mountPath: /etc/secrets
      name: shared-data

  containers:
  - image: nginx
    name: nginx-container
    ports:
    - containerPort: 80
    volumeMounts:
    - mountPath: /usr/share/nginx/html
      name: shared-data

    The example Pod spec (example-k8s-spec.yaml) spins up two containers in vault-agent-example Pod. A vault container which runs Vault Agent as an Init Container. And an nginx container exposing port 80.

    Kubernetes

    NOTE: Set the VAULT_ADDR value to where your Vault server is running. If you are following the Minikube steps, this value should be the EXTERNAL_VAULT_ADDR value you discovered in determine the Vault address.

  5. Create the vault-agent-example Pod defined in example-k8s-spec.yaml.

    $ kubectl apply -f example-k8s-spec.yaml --record

    This takes a minute or so for the Pod to become fully up and running.

»Verification

  1. In another terminal, launch the Minikube dashboard.

    $ minikube dashboard

  2. Click Pods under Workloads to verify that vault-agent-example Pod has been created successfully.

    Kubernetes

    1. Select vault-agent-example to see its details.

    Kubernetes

  3. Port-forward so you can connect to the client from browser.

    $ kubectl port-forward pod/vault-agent-example 8080:80

  4. In a web browser, go to localhost:8080

    Kubernetes

    Notice that the username and password values were successfully read from secret/myapp/config.

  5. Optionally, you can view the HTML source.

    $ kubectl exec -it vault-agent-example --container nginx-container sh

/# cat /usr/share/nginx/html/index.html
  <html>
  <body>
  <p>Some secrets:</p>
  <ul>
  <li><pre>username: appuser</pre></li>
  <li><pre>password: suP3rsec(et!</pre></li>
  </ul>

  </body>
  </html>

»Azure Kubernetes Service Cluster

If you wish to test the Kubernetes auth method against an Azure Kubernetes Service (AKS) cluster instead of Minikube, you can run Terraform to provision an AKS cluster.

First, follow the instruction in the Terraform documentation to create a service principal.

Be sure to add the following Azure Active Directory Graph API permission to your app.

API permissions

Store the credentials as Environment Variables.

$ export ARM_CLIENT_ID="00000000-0000-0000-0000-000000000000"
$ export ARM_CLIENT_SECRET="00000000-0000-0000-0000-000000000000"
$ export ARM_SUBSCRIPTION_ID="00000000-0000-0000-0000-000000000000"
$ export ARM_TENANT_ID="00000000-0000-0000-0000-000000000000"

  1. Set your working directory to where the /identity/vault-agent-k8s-demo/terraform-azure folder is located.

  2. Modify terraform.tfvars.example and provide Azure credentials: client_id and client_secret, and save it as terraform.tfvars.

  1. Execute the Terraform commands to provision a new AKS cluster.

    Pull necessary plugins.

    $ terraform init

    Now, execute the apply command to build a new AKS cluster.

    $ terraform apply -auto-approve

    $ terraform refresh
...

Outputs:

kubernetes_cluster_name = education-xxxxxx-aks
resource_group_name = education-xxxxxx-rg

  2. Now, view the information about the AKS cluster.

    $ kubectl cluster-info

Kubernetes master is running at https://education.hcp.westus2.azmk8s.io:443
Heapster is running at ...
...

    NOTE: Copy the Kubernetes master address (https://education.hcp.westus2.azmk8s.io:443 in this example).

  3. In the /vault-agent-k8s-demo/setup-k8s-auth.sh file, replace Line 48 to point to the AKS cluster address rather than export K8S_HOST=$(minikube ip). Also, replace Line 54 to point to the correct host address.

    Example (setup-k8s-auth.sh):

    ...
# Set K8S_HOST to minikube IP address
# export K8S_HOST=$(minikube ip)
export K8S_HOST="https://education.hcp.westus2.azmk8s.io:443"

# Enable the Kubernetes auth method at the default path ("auth/kubernetes")
vault auth enable kubernetes

# Tell Vault how to communicate with the Kubernetes (Minikube) cluster
# vault write auth/kubernetes/config token_reviewer_jwt="$SA_JWT_TOKEN" kubernetes_host="https://$K8S_HOST:8443" kubernetes_ca_cert="$SA_CA_CRT"
vault write auth/kubernetes/config token_reviewer_jwt="$SA_JWT_TOKEN" kubernetes_host="$K8S_HOST" kubernetes_ca_cert="$SA_CA_CRT"
...

  4. Set the working directory to where scripts are located (/vault-agent-k8s-demo/).

    $ cd ..

  5. Create a service account, vault-auth.

    $ kubectl create serviceaccount vault-auth

  6. Update the vault-auth service account based on the definition in the vault-auth-service-account.yaml file.

    $ kubectl apply --filename vault-auth-service-account.yaml

  7. Setup the Kubernetes auth method on the Vault server.

    $ ./setup-k8s-auth.sh

  8. Resume with the determine the Vault address step and on.

»AKS Kubernetes Dashboard

Click on the Monitor containers in the Azure portal.

Kubernetes

Click the Enable button so that you can monitor the Kubernetes cluster utilization metrics.

»Clean up

When you are done experimenting this demo, execute the following commands to clean up.

Execute the terraform destroy command.

$ terraform destroy -force

Remove the Terraform state files.

$ rm -rf .terraform terraform.tfstate*

»Google Kubernetes Engine Cluster

  1. Set your working directory to where the /identity/vault-agent-k8s-demo/terraform-gcp folder is located.

  2. Modify terraform.tfvars.example and provide GCP credentials: account_file_path and project.

Example:

account_file_path = "/usr/student/gcp/vault-test-project.json"
project  = "vault-test-project"

  1. Execute the Terraform commands to provision a new GKE cluster.

    Pull necessary plugins.

    $ terraform init

    Now, execute the apply command to build a new GKE cluster.

    $ terraform apply -auto-approve

  2. Connect to the GKE cluster.

    $ gcloud container clusters get-credentials $(terraform output gcp_cluster_name) \
        --zone $(terraform output gcp_zone) \
        --project $(terraform output gcp_project)

  3. Get the cluster information.

    $ kubectl cluster-info

Kubernetes master is running at https://198.51.100.24
GLBCDefaultBackend is running at https://198.51.100.24/api/v1/namespaces/...
...

    NOTE: If you don't have the gcloud command line tool installed, follow the online documentation.

  4. Copy the Kubernetes master address (https://198.51.100.24 in this example).

  5. In the /vault-agent-k8s-demo/setup-k8s-auth.sh file, replace Line 48 to point to the GKE cluster address rather than export K8S_HOST=$(minikube ip). Also, replace Line 54 to point to the correct host address.

    Example (setup-k8s-auth.sh):

    ...
# Set K8S_HOST to minikube IP address
# export K8S_HOST=$(minikube ip)
export K8S_HOST="https://198.51.100.24"

# Enable the Kubernetes auth method at the default path ("auth/kubernetes")
vault auth enable kubernetes

# Tell Vault how to communicate with the Kubernetes (Minikube) cluster
# vault write auth/kubernetes/config token_reviewer_jwt="$SA_JWT_TOKEN" kubernetes_host="https://$K8S_HOST:8443" kubernetes_ca_cert="$SA_CA_CRT"
vault write auth/kubernetes/config token_reviewer_jwt="$SA_JWT_TOKEN" kubernetes_host="$K8S_HOST" kubernetes_ca_cert="$SA_CA_CRT"
...

  6. Set the working directory to where scripts are located (/vault-agent-k8s-demo/).

    $ cd ..

  7. Create a service account, vault-auth.

    $ kubectl create serviceaccount vault-auth

  8. Update the vault-auth service account with definition provided in the vault-auth-service-account.yaml file.

    $ kubectl apply --filename vault-auth-service-account.yaml

  9. Setup the Kubernetes auth method on the Vault server.

    $ ./setup-k8s-auth.sh

  10. Resume with the determine the Vault address step and on.

»Help and Reference

Back to Collection
HashiCorp
stdin: is not a tty