Vault Agent with Kubernetes

Nearly all requests to Vault must be accompanied by an authentication token. This includes all API requests, as well as via the Vault CLI and other libraries. If you can securely get the first secret from an originator to a consumer, all subsequent secrets transmitted between this originator and consumer can be authenticated with the trust established by the successful distribution and user of that first secret.

»Challenge

The applications running in a Kubernetes environment is no exception. Luckily, Vault provides Kubernetes auth method to authenticate the clients using a Kubernetes Service Account Token.

However, the client is still responsible for managing the lifecycle of its Vault tokens. Therefore, the next challenge becomes how to manage the lifecycle of tokens in a standard way without having to write custom logic.

»Solution

Vault Agent provides a number of different helper features, specifically addressing the following challenges:

• Automatic authentication
• Secure delivery/storage of tokens
• Lifecycle management of these tokens (renewal & re-authentication)

»Prerequisites

To perform the tasks described in this tutorial, you need:

• Minikube installed
• A running Vault environment reachable from your Kubernetes environment. Refer to the Getting Started tutorial to install Vault. Make sure that your Vault server has been initialized and unsealed

Retrieve the additional configuration by cloning the hashicorp/learn-vault-agent repository from GitHub.

$ git clone https://github.com/hashicorp/learn-vault-agent.git

$ git clone https://github.com/hashicorp/learn-vault-agent.git

This repository contains supporting content for all of the Vault learn tutorials. The content specific to this tutorial can be found in a sub-directory.

Go into the learn-vault-agent/identity/vault-agent-k8s-demo directory.

$ cd learn-vault-agent/vault-agent-k8s-demo

$ cd learn-vault-agent/vault-agent-k8s-demo

»Start a Vault server

To go through this tutorial, start a Vault dev server which listens for requests locally at 0.0.0.0:8200 with root as the root token ID.

$ vault server -dev -dev-root-token-id root -dev-listen-address 0.0.0.0:8200

$ vault server -dev -dev-root-token-id root -dev-listen-address 0.0.0.0:8200

Setting the -dev-listen-address to 0.0.0.0:8200 overrides the default address of a Vault dev server (127.0.0.1:8200) and enables Vault to be addressable by the Kubernetes cluster and its pods because it binds to a shared network.

Export an environment variable for the vault CLI to address the Vault server.

$ export VAULT_ADDR=http://0.0.0.0:8200

$ export VAULT_ADDR=http://0.0.0.0:8200

»Create a service account

1. Start a Kubernetes cluster running in Minikube.

   $ minikube start --driver=docker
   

   $ minikube start --driver=docker
   

   Copy

   Wait a couple of minutes for the minikube environment to become fully available.

   $ minikube status
   
   minikube
   type: Control Plane
   host: Running
   kubelet: Running
   apiserver: Running
   kubeconfig: Configured
   

   $ minikube status
   
   minikube
   type: Control Plane
   host: Running
   kubelet: Running
   apiserver: Running
   kubeconfig: Configured
   

   Copy

2. In Kubernetes, a service account provides an identity for processes that run in a Pod so that the processes can contact the API server. Open the provided vault-auth-service-account.yaml file in your preferred text editor and examine its content for the service account definition to be used for this tutorial.

   ---
   apiVersion: rbac.authorization.k8s.io/v1
   kind: ClusterRoleBinding
   metadata:
     name: role-tokenreview-binding
     namespace: default
   roleRef:
     apiGroup: rbac.authorization.k8s.io
     kind: ClusterRole
     name: system:auth-delegator
   subjects:
   - kind: ServiceAccount
     name: vault-auth
     namespace: default
   

   vault-auth-service-account.yaml

   ---
   apiVersion: rbac.authorization.k8s.io/v1
   kind: ClusterRoleBinding
   metadata:
     name: role-tokenreview-binding
     namespace: default
   roleRef:
     apiGroup: rbac.authorization.k8s.io
     kind: ClusterRole
     name: system:auth-delegator
   subjects:
   - kind: ServiceAccount
     name: vault-auth
     namespace: default
   

3. Create a Kubernetes service account named vault-auth.

   $ kubectl create serviceaccount vault-auth
   

   $ kubectl create serviceaccount vault-auth
   

   Copy

4. Update the vault-auth service account.

   $ kubectl apply --filename vault-auth-service-account.yaml
   

   $ kubectl apply --filename vault-auth-service-account.yaml
   

   Copy

»Configure Kubernetes auth method

1. Create a read-only policy, myapp-kv-ro in Vault.

   $ vault policy write myapp-kv-ro - <<EOF
   path "secret/data/myapp/*" {
       capabilities = ["read", "list"]
   }
   EOF
   

   $ vault policy write myapp-kv-ro - <<EOF
   path "secret/data/myapp/*" {
       capabilities = ["read", "list"]
   }
   EOF
   

   Copy

2. Create some test data at the secret/myapp path.

   $ vault kv put secret/myapp/config \
         username='appuser' \
         password='suP3rsec(et!' \
         ttl='30s'
   

   $ vault kv put secret/myapp/config \
         username='appuser' \
         password='suP3rsec(et!' \
         ttl='30s'
   

   Copy

   Output:

   ====== Secret Path ======
   secret/data/myapp/config
   
   ======= Metadata =======
   Key                Value
   ---                -----
   created_time       2022-03-24T06:09:49.99472Z
   custom_metadata    <nil>
   deletion_time      n/a
   destroyed          false
   version            1
   

   ====== Secret Path ======
   secret/data/myapp/config
   
   ======= Metadata =======
   Key                Value
   ---                -----
   created_time       2022-03-24T06:09:49.99472Z
   custom_metadata    <nil>
   deletion_time      n/a
   destroyed          false
   version            1
   

3. Set the environment variables to point to the running Minikube environment.

   Set the VAULT_SA_NAME environment variable value to the vault-auth service account.

   $ export VAULT_SA_NAME=$(kubectl get sa vault-auth \
       --output jsonpath="{.secrets[*]['name']}")
   

   $ export VAULT_SA_NAME=$(kubectl get sa vault-auth \
       --output jsonpath="{.secrets[*]['name']}")
   

   Copy

   Set the SA_JWT_TOKEN environment variable value to the service account JWT used to access the TokenReview API

   $ export SA_JWT_TOKEN=$(kubectl get secret $VAULT_SA_NAME \
       --output 'go-template={{ .data.token }}' | base64 --decode)
   

   $ export SA_JWT_TOKEN=$(kubectl get secret $VAULT_SA_NAME \
       --output 'go-template={{ .data.token }}' | base64 --decode)
   

   Copy

   Set the SA_CA_CRT environment variable value to the PEM encoded CA cert used to talk to Kubernetes API.

   $ export SA_CA_CRT=$(kubectl config view --raw --minify --flatten \
       --output 'jsonpath={.clusters[].cluster.certificate-authority-data}' | base64 --decode)
   

   $ export SA_CA_CRT=$(kubectl config view --raw --minify --flatten \
       --output 'jsonpath={.clusters[].cluster.certificate-authority-data}' | base64 --decode)
   

   Copy

   Set the K8S_HOST environment variable value to minikube IP address.

   $ export K8S_HOST=$(kubectl config view --raw --minify --flatten \
       --output 'jsonpath={.clusters[].cluster.server}')
   

   $ export K8S_HOST=$(kubectl config view --raw --minify --flatten \
       --output 'jsonpath={.clusters[].cluster.server}')
   

   Copy

4. Now, enable and configure the Kubernetes auth method.

   Enable the Kubernetes auth method at the default path ("auth/kubernetes").

   $ vault auth enable kubernetes
   
   Success! Enabled kubernetes auth method at: kubernetes/
   

   $ vault auth enable kubernetes
   
   Success! Enabled kubernetes auth method at: kubernetes/
   

   Copy

   Tell Vault how to communicate with the Kubernetes (Minikube) cluster.

   $ vault write auth/kubernetes/config \
        token_reviewer_jwt="$SA_JWT_TOKEN" \
        kubernetes_host="$K8S_HOST" \
        kubernetes_ca_cert="$SA_CA_CRT" \
        issuer="https://kubernetes.default.svc.cluster.local"
   

   $ vault write auth/kubernetes/config \
        token_reviewer_jwt="$SA_JWT_TOKEN" \
        kubernetes_host="$K8S_HOST" \
        kubernetes_ca_cert="$SA_CA_CRT" \
        issuer="https://kubernetes.default.svc.cluster.local"
   

   Copy

   Output:

   Success! Data written to: auth/kubernetes/config
   

   Success! Data written to: auth/kubernetes/config
   

   You can validate the issuer name of your Kubernetes cluster using this method.

5. Create a role named, example, that maps the Kubernetes Service Account to Vault policies and default token TTL.

   $ vault write auth/kubernetes/role/example \
        bound_service_account_names=vault-auth \
        bound_service_account_namespaces=default \
        policies=myapp-kv-ro \
        ttl=24h
   

   $ vault write auth/kubernetes/role/example \
        bound_service_account_names=vault-auth \
        bound_service_account_namespaces=default \
        policies=myapp-kv-ro \
        ttl=24h
   

   Copy

   Output:

   Success! Data written to: auth/kubernetes/role/example
   

   Success! Data written to: auth/kubernetes/role/example
   

»Determine the Vault address

A service bound to all networks on the host, as you configured Vault, is addressable by pods within Minikube's cluster by sending requests to the gateway address of the Kubernetes cluster.

1. Start a minikube SSH session.

   $ minikube ssh
   ## ... minikube ssh login
   

   $ minikube ssh
   ## ... minikube ssh login
   

   Copy

2. Within this SSH session, retrieve the value of the Minikube host.

   $ dig +short host.docker.internal
   192.168.65.2
   

   $ dig +short host.docker.internal
   192.168.65.2
   

   Copy

   Docker networking: The host has a changing IP address (or none if you have no network access). We recommend that you connect to the special DNS name host.docker.internal which resolves to the internal IP address used by the host. host.docker.internal. This is for development purpose and will not work in production. For more information, review the documentation for Mac, Windows.

3. Next, retrieve the status of the Vault server to verify network connectivity.

   $ dig +short host.docker.internal | xargs -I{} curl -s http://{}:8200/v1/sys/seal-status
   {
     "type": "shamir",
     "initialized": true,
     "sealed": false,
     "t": 1,
     "n": 1,
     "progress": 0,
     "nonce": "",
     "version": "1.4.1+ent",
     "migration": false,
     "cluster_name": "vault-cluster-3de6c2d3",
     "cluster_id": "10fd177e-d55a-d740-0c54-26268ed86e31",
     "recovery_seal": false,
     "storage_type": "inmem"
   }
   

   $ dig +short host.docker.internal | xargs -I{} curl -s http://{}:8200/v1/sys/seal-status
   {
     "type": "shamir",
     "initialized": true,
     "sealed": false,
     "t": 1,
     "n": 1,
     "progress": 0,
     "nonce": "",
     "version": "1.4.1+ent",
     "migration": false,
     "cluster_name": "vault-cluster-3de6c2d3",
     "cluster_id": "10fd177e-d55a-d740-0c54-26268ed86e31",
     "recovery_seal": false,
     "storage_type": "inmem"
   }
   

   Copy

   The output displays that Vault is initialized and unsealed. This confirms that pods within your cluster are able to reach Vault given that each pod is configured to use the gateway address.

4. Next, exit the Minikube SSH session.

   $ exit
   

   $ exit
   

   Copy

5. Finally, create a variable named EXTERNAL_VAULT_ADDR to capture the Minikube gateway address.

   $ EXTERNAL_VAULT_ADDR=$(minikube ssh "dig +short host.docker.internal" | tr -d '\r')
   

   $ EXTERNAL_VAULT_ADDR=$(minikube ssh "dig +short host.docker.internal" | tr -d '\r')
   

   Copy

6. Verify that the variable contains the IP address you saw when executed in the Minikube shell.

   $ echo $EXTERNAL_VAULT_ADDR
   192.168.65.2
   

   $ echo $EXTERNAL_VAULT_ADDR
   192.168.65.2
   

   Copy

»Optional: Verify the Kubernetes auth method configuration

1. Define a Pod with a container.

   $ cat > devwebapp.yaml <<EOF
   apiVersion: v1
   kind: Pod
   metadata:
     name: devwebapp
     labels:
       app: devwebapp
   spec:
     serviceAccountName: vault-auth
     containers:
       - name: devwebapp
         image: burtlo/devwebapp-ruby:k8s
         env:
           - name: VAULT_ADDR
             value: "http://$EXTERNAL_VAULT_ADDR:8200"
   EOF
   

   $ cat > devwebapp.yaml <<EOF
   apiVersion: v1
   kind: Pod
   metadata:
     name: devwebapp
     labels:
       app: devwebapp
   spec:
     serviceAccountName: vault-auth
     containers:
       - name: devwebapp
         image: burtlo/devwebapp-ruby:k8s
         env:
           - name: VAULT_ADDR
             value: "http://$EXTERNAL_VAULT_ADDR:8200"
   EOF
   

   Copy

   The Pod is named devwebapp and runs with the vault-auth service account.

2. Create the devwebapp pod in the default namespace

   $ kubectl apply --filename devwebapp.yaml --namespace default
   

   $ kubectl apply --filename devwebapp.yaml --namespace default
   

   Copy

3. Display all the pods in the default namespace.

   $ kubectl get pods
   NAME        READY   STATUS    RESTARTS   AGE
   devwebapp   1/1     Running   0          77s
   

   $ kubectl get pods
   NAME        READY   STATUS    RESTARTS   AGE
   devwebapp   1/1     Running   0          77s
   

   Copy

   Wait until the devwebapp pod is running and ready (1/1).

4. Start an interactive shell session on the devwebapp pod.

   $ kubectl exec --stdin=true --tty=true devwebapp -- /bin/sh
   #
   

   $ kubectl exec --stdin=true --tty=true devwebapp -- /bin/sh
   #
   

   Copy

   Your system prompt is replaced with a new prompt #.

   Note: the prompt within this section is shown as $ but the commands are intended to be executed within this interactive shell on the devwebapp container.

5. Set KUBE_TOKEN to the service account token.

   $ KUBE_TOKEN=$(cat /var/run/secrets/kubernetes.io/serviceaccount/token)
   

   $ KUBE_TOKEN=$(cat /var/run/secrets/kubernetes.io/serviceaccount/token)
   

   Copy

6. Authenticate with Vault through the example role with the KUBE_TOKEN.

   $ curl --request POST \
          --data '{"jwt": "'"$KUBE_TOKEN"'", "role": "example"}' \
          $VAULT_ADDR/v1/auth/kubernetes/login
   

   $ curl --request POST \
          --data '{"jwt": "'"$KUBE_TOKEN"'", "role": "example"}' \
          $VAULT_ADDR/v1/auth/kubernetes/login
   

   Copy

   Example output:

   {
     ...
     "auth": {
       "client_token": "s.7cH83AFIdmXXYKsPsSbeESpp",
       "accessor": "8bmYWFW5HtwDHLAoxSiuMZRh",
       "policies": [
         "default",
         "myapp-kv-ro"
       ],
       "token_policies": [
         "default",
         "myapp-kv-ro"
       ],
       "metadata": {
         "role": "example",
         "service_account_name": "vault-auth",
         "service_account_namespace": "default",
         "service_account_secret_name": "vault-auth-token-vqqlp",
         "service_account_uid": "adaca842-f2a7-11e8-831e-080027b85b6a"
       },
       "lease_duration": 86400,
       "renewable": true,
       "entity_id": "2c4624f1-29d6-972a-fb27-729b50dd05e2",
       "token_type": "service"
     }
   }
   

   {
     ...
     "auth": {
       "client_token": "s.7cH83AFIdmXXYKsPsSbeESpp",
       "accessor": "8bmYWFW5HtwDHLAoxSiuMZRh",
       "policies": [
         "default",
         "myapp-kv-ro"
       ],
       "token_policies": [
         "default",
         "myapp-kv-ro"
       ],
       "metadata": {
         "role": "example",
         "service_account_name": "vault-auth",
         "service_account_namespace": "default",
         "service_account_secret_name": "vault-auth-token-vqqlp",
         "service_account_uid": "adaca842-f2a7-11e8-831e-080027b85b6a"
       },
       "lease_duration": 86400,
       "renewable": true,
       "entity_id": "2c4624f1-29d6-972a-fb27-729b50dd05e2",
       "token_type": "service"
     }
   }
   

   Notice that client_token is successfully generated and myapp-kv-ro policy is attached with the token. The metadata displays that its service account name (service_account_name) is vault-auth.

7. Lastly, exit the pod.

   $ exit
   

   $ exit
   

   Copy

»Start Vault Agent with Auto-Auth

Now that you have verified that the Kubernetes auth method has been configured on the Vault server, it is time to spin up a client Pod which leverages Vault Agent to automatically authenticate with Vault and retrieve a client token.

1. First, open the provided configmap.yaml file in your preferred text editor and review its content.

   apiVersion: v1
   data:
     vault-agent-config.hcl: |
       # Comment this out if running as sidecar instead of initContainer
       exit_after_auth = true
   
       pid_file = "/home/vault/pidfile"
   
       auto_auth {
           method "kubernetes" {
               mount_path = "auth/kubernetes"
               config = {
                   role = "example"
               }
           }
   
           sink "file" {
               config = {
                   path = "/home/vault/.vault-token"
               }
           }
       }
   
       template {
       destination = "/etc/secrets/index.html"
       contents = <<EOT
       <html>
       <body>
       <p>Some secrets:</p>
       {{- with secret "secret/data/myapp/config" }}
       <ul>
       <li><pre>username: {{ .Data.data.username }}</pre></li>
       <li><pre>password: {{ .Data.data.password }}</pre></li>
       </ul>
       {{ end }}
       </body>
       </html>
       EOT
       }
   kind: ConfigMap
   metadata:
     name: example-vault-agent-config
     namespace: default
   

   configmap.yaml

   1 2 3 4 5 6 7 8 9 10111213141516171819202122232425262728293031323334353637383940414243apiVersion: v1
   data:
     vault-agent-config.hcl: |
       # Comment this out if running as sidecar instead of initContainer
       exit_after_auth = true
   
       pid_file = "/home/vault/pidfile"
   
       auto_auth {
           method "kubernetes" {
               mount_path = "auth/kubernetes"
               config = {
                   role = "example"
               }
           }
   
           sink "file" {
               config = {
                   path = "/home/vault/.vault-token"
               }
           }
       }
   
       template {
       destination = "/etc/secrets/index.html"
       contents = <<EOT
       <html>
       <body>
       <p>Some secrets:</p>
       {{- with secret "secret/data/myapp/config" }}
       <ul>
       <li><pre>username: {{ .Data.data.username }}</pre></li>
       <li><pre>password: {{ .Data.data.password }}</pre></li>
       </ul>
       {{ end }}
       </body>
       </html>
       EOT
       }
   kind: ConfigMap
   metadata:
     name: example-vault-agent-config
     namespace: default
   

   This creates a Vault Agent configuration file, vault-agent-config.hcl. Notice that the Vault Agent Auto-Auth (auto_auth block) is configured to use the kubernetes auth method enabled at the auth/kubernetes path on the Vault server. The Vault Agent will use the example role which you created in Step 2.

   The sink block specifies the location on disk where to write tokens. Vault Agent Auto-Auth sink can be configured multiple times if you want Vault Agent to place the token into multiple locations. In this example, the sink is set to /home/vault/.vault-token.

   Finally, the template block creates a templated file which retrieves username and password values at the secret/data/myapp/config path.

2. Create a ConfigMap containing a Vault Agent configuration.

   $ kubectl create --filename configmap.yaml
   

   $ kubectl create --filename configmap.yaml
   

   Copy

3. View the created ConfigMap.

   $ kubectl get configmap example-vault-agent-config --output yaml
   

   $ kubectl get configmap example-vault-agent-config --output yaml
   

   Copy

4. An example Pod spec file is provided. Review the provided example Pod spec file, example-k8s-spec.yaml.

   apiVersion: v1
   kind: Pod
   metadata:
     name: vault-agent-example
     namespace: default
   spec:
     serviceAccountName: vault-auth
   
     volumes:
       - configMap:
           items:
             - key: vault-agent-config.hcl
               path: vault-agent-config.hcl
           name: example-vault-agent-config
         name: config
       - emptyDir: {}
         name: shared-data
   
     initContainers:
       - args:
           - agent
           - -config=/etc/vault/vault-agent-config.hcl
           - -log-level=debug
         env:
           - name: VAULT_ADDR
             value: http://EXTERNAL_VAULT_ADDR:8200
         image: vault
         name: vault-agent
         volumeMounts:
           - mountPath: /etc/vault
             name: config
           - mountPath: /etc/secrets
             name: shared-data
   
     containers:
       - image: nginx
         name: nginx-container
         ports:
           - containerPort: 80
         volumeMounts:
           - mountPath: /usr/share/nginx/html
             name: shared-data
   

   example-k8s-spec.yaml

   apiVersion: v1
   kind: Pod
   metadata:
     name: vault-agent-example
     namespace: default
   spec:
     serviceAccountName: vault-auth
   
     volumes:
       - configMap:
           items:
             - key: vault-agent-config.hcl
               path: vault-agent-config.hcl
           name: example-vault-agent-config
         name: config
       - emptyDir: {}
         name: shared-data
   
     initContainers:
       - args:
           - agent
           - -config=/etc/vault/vault-agent-config.hcl
           - -log-level=debug
         env:
           - name: VAULT_ADDR
             value: http://EXTERNAL_VAULT_ADDR:8200
         image: vault
         name: vault-agent
         volumeMounts:
           - mountPath: /etc/vault
             name: config
           - mountPath: /etc/secrets
             name: shared-data
   
     containers:
       - image: nginx
         name: nginx-container
         ports:
           - containerPort: 80
         volumeMounts:
           - mountPath: /usr/share/nginx/html
             name: shared-data
   

   The example Pod spec (example-k8s-spec.yaml) spins up two containers in vault-agent-example pod. A vault container which runs Vault Agent as an Init Container. And an nginx container exposing port 80.

   

   The Vault address, VAULT_ADDR, is set to a placeholder value EXTERNAL_VAULT_ADDR.

5. Generate the Pod spec with EXTERNAL_VAULT_ADDR variable value in its place.

   $ cat example-k8s-spec.yaml | \
      sed -e s/"EXTERNAL_VAULT_ADDR"/"$EXTERNAL_VAULT_ADDR"/ \
      > vault-agent-example.yaml
   

   $ cat example-k8s-spec.yaml | \
      sed -e s/"EXTERNAL_VAULT_ADDR"/"$EXTERNAL_VAULT_ADDR"/ \
      > vault-agent-example.yaml
   

   Copy

6. Create the vault-agent-example pod defined in vault-agent-example.yaml.

   $ kubectl apply --filename vault-agent-example.yaml --record
   

   $ kubectl apply --filename vault-agent-example.yaml --record
   

   Copy

   This takes a minute or so for the pod to become fully up and running.

»Verification

1. In another terminal, launch the Minikube dashboard.

   $ minikube dashboard
   

   $ minikube dashboard
   

   Copy

2. Click Pods under Workloads to verify that vault-agent-example pod has been created successfully.

   

3. Select vault-agent-example to see its details.

   

4. In another terminal, port forward all requests made to http://localhost:8080 to port 80 on the vault-agent-example pod.

   $ kubectl port-forward pod/vault-agent-example 8080:80
   

   $ kubectl port-forward pod/vault-agent-example 8080:80
   

   Copy

5. In a web browser, go to localhost:8080

   

   Notice that the username and password values were successfully read from secret/myapp/config.

6. Optionally, you can view the HTML source.

   $ kubectl exec -it vault-agent-example --container nginx-container -- cat /usr/share/nginx/html/index.html
   <html>
   <body>
   <p>Some secrets:</p>
   <ul>
   <li><pre>username: appuser</pre></li>
   <li><pre>password: suP3rsec(et!</pre></li>
   </ul>
   
   </body>
   </html>
   

   $ kubectl exec -it vault-agent-example --container nginx-container -- cat /usr/share/nginx/html/index.html
   <html>
   <body>
   <p>Some secrets:</p>
   <ul>
   <li><pre>username: appuser</pre></li>
   <li><pre>password: suP3rsec(et!</pre></li>
   </ul>
   
   </body>
   </html>
   

   Copy

»Azure Kubernetes Service Cluster

If you wish to test the Kubernetes auth method against an Azure Kubernetes Service (AKS) cluster instead of Minikube, you can run Terraform to provision an AKS cluster.

First, follow the instruction in the Terraform documentation to create a service principal.

Be sure to add the following Azure Active Directory Graph API permission to your app.

Store the credentials as Environment Variables.

$ export ARM_CLIENT_ID="00000000-0000-0000-0000-000000000000"
$ export ARM_CLIENT_SECRET="00000000-0000-0000-0000-000000000000"
$ export ARM_SUBSCRIPTION_ID="00000000-0000-0000-0000-000000000000"
$ export ARM_TENANT_ID="00000000-0000-0000-0000-000000000000"

$ export ARM_CLIENT_ID="00000000-0000-0000-0000-000000000000"
$ export ARM_CLIENT_SECRET="00000000-0000-0000-0000-000000000000"
$ export ARM_SUBSCRIPTION_ID="00000000-0000-0000-0000-000000000000"
$ export ARM_TENANT_ID="00000000-0000-0000-0000-000000000000"

• Subscription ID: Navigate to the Subscriptions blade within the Azure Portal and copy the SUBSCRIPTION ID

• Tenant ID: Navigate to the Azure Active Directory > Properties in the Azure Portal, and copy the Directory ID which is your tenant ID

• Client ID: Same as the Application ID

• Client secret: The password (credential) set on your application

1. Set your working directory to where the learn-vault-agent/vault-agent-k8s-demo/terraform-azure folder is located.

2. Modify terraform.tfvars.example and provide Azure credentials: client_id and client_secret, and save it as terraform.tfvars.

   Client ID: Same as the Application ID

   Client secret: The password (credential) set on your application

3. Execute the Terraform commands to provision a new AKS cluster.

   Pull necessary plugins.

   $ terraform init
   

   $ terraform init
   

   Copy

   Now, execute the apply command to build a new AKS cluster.

   $ terraform apply -auto-approve
   

   $ terraform apply -auto-approve
   

   Copy

   NOTE: If you received Resource group 'education-XXXXX-rg' could not be found error, run terraform refresh command.

   $ terraform refresh
   ...
   
   Outputs:
   
   kubernetes_cluster_name = education-xxxxxx-aks
   resource_group_name = education-xxxxxx-rg
   

   $ terraform refresh
   ...
   
   Outputs:
   
   kubernetes_cluster_name = education-xxxxxx-aks
   resource_group_name = education-xxxxxx-rg
   

   Copy

4. Now, view the information about the AKS cluster.

   $ kubectl cluster-info
   
   Kubernetes master is running at https://education.hcp.westus2.azmk8s.io:443
   Heapster is running at ...
   ...
   

   $ kubectl cluster-info
   
   Kubernetes master is running at https://education.hcp.westus2.azmk8s.io:443
   Heapster is running at ...
   ...
   

   Copy

   NOTE: Copy the Kubernetes master address (https://education.hcp.westus2.azmk8s.io:443 in this example).

5. In the /vault-agent-k8s-demo/setup-k8s-auth.sh file, replace Line 48 to point to the AKS cluster address rather than export K8S_HOST=$(minikube ip). Also, replace Line 54 to point to the correct host address.

   Example:

   ...
   # Set K8S_HOST to minikube IP address
   # export K8S_HOST=$(minikube ip)
   export K8S_HOST="https://education.hcp.westus2.azmk8s.io:443"
   
   # Enable the Kubernetes auth method at the default path ("auth/kubernetes")
   vault auth enable kubernetes
   
   # Tell Vault how to communicate with the Kubernetes (Minikube) cluster
   # vault write auth/kubernetes/config token_reviewer_jwt="$SA_JWT_TOKEN" kubernetes_host="https://$K8S_HOST:8443" kubernetes_ca_cert="$SA_CA_CRT"
   vault write auth/kubernetes/config token_reviewer_jwt="$SA_JWT_TOKEN" kubernetes_host="$K8S_HOST" kubernetes_ca_cert="$SA_CA_CRT"
   ...
   

   setup-k8s-auth.sh

   ...
   # Set K8S_HOST to minikube IP address
   # export K8S_HOST=$(minikube ip)
   export K8S_HOST="https://education.hcp.westus2.azmk8s.io:443"
   
   # Enable the Kubernetes auth method at the default path ("auth/kubernetes")
   vault auth enable kubernetes
   
   # Tell Vault how to communicate with the Kubernetes (Minikube) cluster
   # vault write auth/kubernetes/config token_reviewer_jwt="$SA_JWT_TOKEN" kubernetes_host="https://$K8S_HOST:8443" kubernetes_ca_cert="$SA_CA_CRT"
   vault write auth/kubernetes/config token_reviewer_jwt="$SA_JWT_TOKEN" kubernetes_host="$K8S_HOST" kubernetes_ca_cert="$SA_CA_CRT"
   ...
   

6. Set the working directory to where scripts are located (learn-vault-agent/vault-agent-k8s-demo).

   $ cd ..
   

   $ cd ..
   

   Copy

7. Create a service account, vault-auth.

   $ kubectl create serviceaccount vault-auth
   

   $ kubectl create serviceaccount vault-auth
   

   Copy

8. Update the vault-auth service account based on the definition in the vault-auth-service-account.yaml file.

   $ kubectl apply --filename vault-auth-service-account.yaml
   

   $ kubectl apply --filename vault-auth-service-account.yaml
   

   Copy

9. Setup the Kubernetes auth method on the Vault server.

   $ ./setup-k8s-auth.sh
   

   $ ./setup-k8s-auth.sh
   

   Copy

10. Resume with the determine the Vault address step and on.

»AKS Kubernetes Dashboard

Click on the Monitor containers in the Azure portal.

Click the Enable button so that you can monitor the Kubernetes cluster utilization metrics.

»Clean up

When you are done experimenting this demo, execute the following commands to clean up.

Execute the terraform command.

$ terraform apply -destroy -auto-approve

$ terraform apply -destroy -auto-approve

Remove the Terraform state files.

$ rm *tfstate*

$ rm *tfstate*

»Google Kubernetes Engine Cluster

1. Set your working directory to where the learn-vault-agent/vault-agent-k8s-demo/terraform-gcp folder is located.

2. Modify terraform.tfvars.example and provide GCP credentials: account_file_path and project, and save it as terraform.tfvars.

   Example:

   account_file_path = "/usr/student/gcp/vault-test-project.json"
   project  = "vault-test-project"
   

   terraform.tfvars

   account_file_path = "/usr/student/gcp/vault-test-project.json"
   project  = "vault-test-project"
   

3. Execute the Terraform commands to provision a new GKE cluster.

   Pull necessary plugins.

   $ terraform init
   

   $ terraform init
   

   Copy

   Now, execute the apply command to build a new GKE cluster.

   $ terraform apply -auto-approve
   

   $ terraform apply -auto-approve
   

   Copy

4. Connect to the GKE cluster.

   $ gcloud container clusters get-credentials $(terraform output gcp_cluster_name) \
           --zone $(terraform output gcp_zone) \
           --project $(terraform output gcp_project)
   

   $ gcloud container clusters get-credentials $(terraform output gcp_cluster_name) \
           --zone $(terraform output gcp_zone) \
           --project $(terraform output gcp_project)
   

   Copy

5. Get the cluster information.

   $ kubectl cluster-info
   
   Kubernetes master is running at https://198.51.100.24
   GLBCDefaultBackend is running at https://198.51.100.24/api/v1/namespaces/...
   ...
   

   $ kubectl cluster-info
   
   Kubernetes master is running at https://198.51.100.24
   GLBCDefaultBackend is running at https://198.51.100.24/api/v1/namespaces/...
   ...
   

   Copy

   NOTE: If you don't have the gcloud command line tool installed, follow the online documentation.

6. Copy the Kubernetes master address (https://198.51.100.24 in this example).

7. In the /vault-agent-k8s-demo/setup-k8s-auth.sh file, replace Line 48 to point to the GKE cluster address rather than export K8S_HOST=$(minikube ip). Also, replace Line 54 to point to the correct host address.

   Example:

   ...
   # Set K8S_HOST to minikube IP address
   # export K8S_HOST=$(minikube ip)
   export K8S_HOST="https://198.51.100.24"
   
   # Enable the Kubernetes auth method at the default path ("auth/kubernetes")
   vault auth enable kubernetes
   
   # Tell Vault how to communicate with the Kubernetes (Minikube) cluster
   # vault write auth/kubernetes/config token_reviewer_jwt="$SA_JWT_TOKEN" kubernetes_host="https://$K8S_HOST:8443" kubernetes_ca_cert="$SA_CA_CRT"
   vault write auth/kubernetes/config token_reviewer_jwt="$SA_JWT_TOKEN" kubernetes_host="$K8S_HOST" kubernetes_ca_cert="$SA_CA_CRT"
   ...
   

   setup-k8s-auth.sh

   ...
   # Set K8S_HOST to minikube IP address
   # export K8S_HOST=$(minikube ip)
   export K8S_HOST="https://198.51.100.24"
   
   # Enable the Kubernetes auth method at the default path ("auth/kubernetes")
   vault auth enable kubernetes
   
   # Tell Vault how to communicate with the Kubernetes (Minikube) cluster
   # vault write auth/kubernetes/config token_reviewer_jwt="$SA_JWT_TOKEN" kubernetes_host="https://$K8S_HOST:8443" kubernetes_ca_cert="$SA_CA_CRT"
   vault write auth/kubernetes/config token_reviewer_jwt="$SA_JWT_TOKEN" kubernetes_host="$K8S_HOST" kubernetes_ca_cert="$SA_CA_CRT"
   ...
   

8. Set the working directory to where scripts are located (learn-vault-agent/vault-agent-k8s-demo).

   $ cd ..
   

   $ cd ..
   

   Copy

9. Create a service account, vault-auth.

   $ kubectl create serviceaccount vault-auth
   

   $ kubectl create serviceaccount vault-auth
   

   Copy

10. Update the vault-auth service account with definition provided in the vault-auth-service-account.yaml file.

    $ kubectl apply --filename vault-auth-service-account.yaml
    

    $ kubectl apply --filename vault-auth-service-account.yaml
    

    Copy

11. Setup the Kubernetes auth method on the Vault server.

    $ ./setup-k8s-auth.sh
    

    $ ./setup-k8s-auth.sh
    

    Copy

12. Resume with the determine the Vault address step and on.

»Help and reference

• Blog post: Why Use the Vault Agent for Secrets Management?
• Video: Streamline Secrets Management with Vault Agent and Vault 0.11
• Secure Introduction of Vault Clients
• Vault Agent Auto-Auth
• Kubernetes Auth Method
• Kubernetes Auth Method (API)
• Vault Kubernetes Workshop on Google Cloud Platform
• Kubernetes Tutorials
• Consul Template

»Other Vault Agent tutorials:

• Vault Agent Templates
• Vault Agent Caching
• Vault Agent Windows Service
• Read Secrets From Vault Using Vault Agent
• Using HashiCorp Vault Agent with .NET Core