Access Management

Vault Agent Caching

Time to Complete
5 minutes
Level
Implementation
Products Used
Vault
Terraform

Nearly all requests to Vault must be accompanied by a valid token which means that Vault clients must first authenticate with Vault and acquire a token. Secure Introduction of Vault Clients guide talked about the approaches to solve this secret zero problem.

Vault Agent with AWS and Vault Agent with Kubernetes guides walked through the Vault Agent's Auto-Auth.

Vault Agent

Challenge & Solution

Depending on the location of your Vault clients and its secret access frequency, you may face some scaling or latency challenge. Even with Vault Performance Replication enabled, the pressure on the storage backend increases as the number of token or lease generation requests increase. Vault 1.0 introduced batch tokens as a solution to relieve some pressure on the storage backend. By design, batch tokens do not support the same level of flexibility and features as service tokens. Therefore, if you need an orphan token for example, you would need service tokens.

To increase the availability of tokens and secrets to the clients, Vault Agent introduced the Caching function.

Vault Agent Caching

Vault Agent Caching can cache the tokens and leased secrets proxied through the agent which includes the auto-auth token. This allows for easier access to Vault secrets for edge applications, reduces the I/O burden for basic secrets access for Vault clusters, and allows for secure local access to leased secrets for the life of a valid token.

Prerequisites

To complete this section of the guide, you will need an AWS account and associated credentials that allow for the creation of resources.

Download demo assets

Clone or download the demo assets from the hashicorp/vault-guides GitHub repository to perform the steps described in this guide.

Steps

You are going to perform the following steps:

  1. Provision the Cloud Resources
  2. Configure AWS IAM Auth Method
  3. Start Vault Agent
  4. Request dynamic secrets and tokens via agent
  5. Cache Evictions
  6. Clean up

Step 1: Provision the Cloud Resources

Be sure to set your working directory to where the /identity/vault-agent-caching/terraform-aws was downloaded.

The working directory should contain the provided Terraform files: Assets

NOTE: The example Terraform in this repository is created for the demo purpose.

  1. Set your AWS credentials as environment variables:

    $ export AWS_ACCESS_KEY_ID = "<YOUR_AWS_ACCESS_KEY_ID>"
$ export AWS_SECRET_ACCESS_KEY = "<YOUR_AWS_SECRET_ACCESS_KEY>"

  2. Create a file named terraform.tfvars and specify the key_name. In addition, specify the variable values (variables.tf) which you wish to overwrite its default. (Use the provided terraform.tfvars.example as a base.)

    Example terrafrom.tfvars:

    # SSH key name to access EC2 instances (should already exist) on the AWS region
key_name = "vault-test"

# All resources will be tagged with this
environment_name = "va-demo"

  3. Perform a terraform init to pull down the necessary provider resources. Then terraform plan to verify your changes and the resources that will be created. If all looks good, then perform a terraform apply to provision the resources.

    $ terraform init

$ terraform plan

$ terraform apply -auto-approve
...
Apply complete! Resources: 23 added, 0 changed, 0 destroyed.

Outputs:

endpoints =
Vault Server IP (public):  54.219.129.15
Vault Server IP (private): 10.0.101.50

For example:
   ssh -i vault-test.pem ubuntu@54.219.129.15

Vault Client IP (public):  54.183.212.51
Vault Client IP (private): 10.0.101.209

For example:
   ssh -i vault-test.pem ubuntu@54.183.212.51

    The Terraform output will display the public IP address to SSH into your Vault server and client instances.

Step 2: Configure AWS IAM Auth Method

In this step, you will configure Vault to allow AWS IAM authentication from specific IAM roles.

  1. SSH into the Vault Server instance.

    Output

    ssh -i <path_to_key> ubuntu@<public_ip_of_server>
...
Are you sure you want to continue connecting (yes/no)? yes

    When you are prompted, enter "yes" to continue.

  2. Run the vault operator init command to initialize the Vault server:

    $ vault operator init
...

Initial Root Token: s.20JnHBY66EKTj9zyR6SjTMNq

Success! Vault is initialized
...

  3. Copy the Initial Root Token value.

  4. Log into Vault using the generated initial root token:

    $ vault login s.20JnHBY66EKTj9zyR6SjTMNq

  5. Examine and then execute the /home/ubuntu/aws_auth.sh script.

    # Review the contnts of aws_auth.sh script
$ cat aws_auth.sh

# Execute the script
$ ./aws_auth.sh
Success! Enabled the kv secrets engine at: kv/
Success! Uploaded policy: myapp
Success! Enabled aws auth method at: aws/
Success! Data written to: auth/aws/config/client
Success! Data written to: auth/aws/role/app-role
Success! Enabled userpass auth method at: userpass/
Success! Data written to: auth/userpass/users/student

    This script enables key/value v1 secrets engine at kv, create myapp policy, enables aws auth method and create a role named app-role. Also, userpass is enabled and created a user (student) with myapp policy attached.

  6. Check the myapp policy.

    $ vault policy read myapp

path "kv/*" {
  capabilities = ["create", "read", "update", "delete"]
}
path "aws/creds/*" {
  capabilities = ["read", "update"]
}
path "sys/leases/*" {
  capabilities = ["create", "update"]
}
path "auth/token/*" {
  capabilities = ["create", "update"]
}

  7. Examine and then execute the /home/ubuntu/aws_secrets.sh script.

    # Review the contents of the aws_secrets.sh script
$ cat aws_secrets.sh

# Execute the script
$ ./aws_secrets.sh
Success! Enabled the aws secrets engine at: aws/
Success! Data written to: aws/config/root
Success! Data written to: aws/config/lease
Success! Data written to: aws/roles/readonly

    This script enables aws secrets engine, configures it and set the generated secret's lease to 1 hour and lease max to 24 hours. It also configures readonly role which is mapped to is mapped to arn:aws:iam::aws:policy/AmazonS3ReadOnlyAccess IAM policy.

Step 3: Start Vault Agent

  1. Now, SSH into the Vault Client instance.

    Output

    ssh -i <path_to_key> ubuntu@<public_ip_of_client>
...
Are you sure you want to continue connecting (yes/no)? yes

    When you are prompted, enter "yes" to continue.

  2. Explore the Vault Agent configuration file, /home/ubuntu/vault-agent.hcl

    $ cat vault-agent.hcl

exit_after_auth = false
pid_file = "./pidfile"

auto_auth {
   method "aws" {
       mount_path = "auth/aws"
       config = {
           type = "iam"
           role = "app-role"
       }
   }

   sink "file" {
       config = {
           path = "/home/ubuntu/vault-token-via-agent"
       }
   }
}

cache {
   use_auto_auth_token = true
}

listener "tcp" {
   address = "127.0.0.1:8200"
   tls_disable = true
}

vault {
   address = "http://<vault-server-host>:8200"
}

  3. Execute the following command to start the Vault Agent with log level set to debug.

    $ vault agent -config=/home/ubuntu/vault-agent.hcl -log-level=debug

==> Vault server started! Log data will stream in below:

==> Vault agent configuration:

           Api Address 1: http://127.0.0.1:8200
                     Cgo: disabled
               Log Level: debug
                 Version: Vault v1.1.0
             Version Sha: 30f07c76e1ea0551f28f5b8f7537f0de49d27f80

2019-03-06T03:12:47.616Z [INFO]  sink.file: creating file sink
2019-03-06T03:12:47.616Z [INFO]  sink.file: file sink configured: path=/home/ubuntu/vault-token-via-agent
2019-03-06T03:12:47.617Z [DEBUG] cache: auto-auth token is allowed to be used; configuring inmem sink
2019-03-06T03:12:47.617Z [INFO]  cache: starting listener: addr=127.0.0.1:8200
2019-03-06T03:12:47.618Z [INFO]  auth.handler: starting auth handler
2019-03-06T03:12:47.618Z [INFO]  auth.handler: authenticating
2019-03-06T03:12:47.618Z [INFO]  sink.server: starting sink server
2019-03-06T03:12:47.883Z [INFO]  auth.handler: authentication successful, sending token to sinks
2019-03-06T03:12:47.883Z [INFO]  auth.handler: starting renewal process
2019-03-06T03:12:47.883Z [INFO]  sink.file: token written: path=/home/ubuntu/vault-token-via-agent
2019-03-06T03:12:47.883Z [DEBUG] cache.leasecache: storing auto-auth token into the cache
2019-03-06T03:12:47.887Z [INFO]  auth.handler: renewed auth token

  4. Open a second SSH terminal into the client machine.

  5. Verify the client token stored in /home/ubuntu/vault-token-via-agent.

    # Verify that a token has been written
$ more vault-token-via-agent

# Check the details about the token (e.g. attached policies, TTL, etc.)
$ VAULT_TOKEN="$(cat vault-token-via-agent)" vault token lookup

  6. Set VAULT_AGENT_ADDR environment variable.

    $ export VAULT_AGENT_ADDR="http://127.0.0.1:8200"

Step 4: Request dynamic secrets and tokens via agent

CLI command / API call using cURL

CLI command

  1. Execute the following command to request an AWS credential.

    $ vault read aws/creds/readonly

Key                Value
---                -----
lease_id           aws/creds/readonly/oWBfLT2nIhXq9ZgWaC2e1fHy
lease_duration     1h
lease_renewable    true
access_key         AKIAJMRXXXXXXXXMAGIA
secret_key         swNXag6aVaultHappyYayAaivQ1iwMBaKCvuDoI
security_token     <nil>

  2. See the logs in the terminal where Vault Agent is running.

    ...
[INFO]  cache: received request: path=/v1/aws/creds/readonly method=GET
[DEBUG] cache: using auto auth token: path=/v1/aws/creds/readonly method=GET
[DEBUG] cache.leasecache: forwarding request: path=/v1/aws/creds/readonly method=GET
[INFO]  cache.apiproxy: forwarding request: path=/v1/aws/creds/readonly method=GET
[DEBUG] cache.leasecache: processing lease response: path=/v1/aws/creds/readonly method=GET
[DEBUG] cache.leasecache: storing response into the cache: path=/v1/aws/creds/readonly method=GET
[DEBUG] cache.leasecache: initiating renewal: path=/v1/aws/creds/readonly method=GET
[DEBUG] cache.leasecache: secret renewed: path=/v1/aws/creds/readonly

    The log indicates that auto-auth token was used to connect Vault. The received request was forwarded to the Vault server, and the returned response was cached (find an entry storing response into the cache in the log).

  3. Re-run the vault command to study the agent behavior.

    $ vault read aws/creds/readonly

Key                Value
---                -----
lease_id           aws/creds/readonly/oWBfLT2nIhXq9ZgWaC2e1fHy
lease_duration     1h
lease_renewable    true
access_key         AKIAJMRXXXXXXXXMAGIA
secret_key         swNXag6aVaultHappyYayAaivQ1iwMBaKCvuDoI
security_token     <nil>

    The agent log indicates:

    [INFO]  cache: received request: path=/v1/aws/creds/readonly method=GET
[DEBUG] cache: using auto auth token: path=/v1/aws/creds/readonly method=GET
[DEBUG] cache.leasecache: returning cached response: path=/v1/aws/creds/readonly

  4. Log in as a user, student via agent whose password is "pAssw0rd":

    # Login with username 'student' and password is "pAssw0rd"
$ vault login -method=userpass username="student" password="pAssw0rd"

Success! You are now authenticated. The token information displayed below
is already stored in the token helper. You do NOT need to run "vault login"
again. Future Vault requests will automatically use this token.

Key                    Value
---                    -----
token                  s.AKsRLji9mPq4xNZP1UIJggk6
token_accessor         lU69kHGAPtYMoQndeCSoYuhs
token_duration         48h
token_renewable        true
token_policies         ["default" "myapp"]
...

# Store the acquired token in VAULT_TOKEN environment variable
$ export VAULT_TOKEN="s.AKsRLji9mPq4xNZP1UIJggk6"

    Examine the agent log in the other terminal:

    ...
[INFO]  cache: received request: path=/v1/auth/userpass/login/student method=POST
[DEBUG] cache: using auto auth token: path=/v1/auth/userpass/login/student method=POST
[DEBUG] cache.leasecache: forwarding request: path=/v1/auth/userpass/login/student method=POST
[INFO]  cache.apiproxy: forwarding request: path=/v1/auth/userpass/login/student method=POST
[DEBUG] cache.leasecache: processing auth response: path=/v1/auth/userpass/login/student method=POST
[DEBUG] cache.leasecache: storing response into the cache: path=/v1/auth/userpass/login/student method=POST
[DEBUG] cache.leasecache: initiating renewal: path=/v1/auth/userpass/login/student method=POST
[DEBUG] cache.leasecache: secret renewed: path=/v1/auth/userpass/login/student
...

    The log should indicate that the acquired token for user student is successfully cached ([DEBUG] cache.leasecache: storing response into the cache). This is because the login request was sent through the agent.

  5. Create a new token with TTL set to 12 minutes:

    $ vault token create -ttl=12m

Key                  Value
---                  -----
token                s.e1h5wPbZ7i6FLCjPDqpsfDBG
token_accessor       y8K7xJr15fFjSowWW1t3VERT
token_duration       12m
...

    Examine the agent log in the other terminal:

    ...
[INFO]  cache: received request: path=/v1/auth/token/create method=POST
[DEBUG] cache.leasecache: forwarding request: path=/v1/auth/token/create method=POST
[INFO]  cache.apiproxy: forwarding request: path=/v1/auth/token/create method=POST
[DEBUG] cache.leasecache: processing auth response: path=/v1/auth/token/create method=POST
[DEBUG] cache.leasecache: setting parent context: path=/v1/auth/token/create method=POST
[DEBUG] cache.leasecache: storing response into the cache: path=/v1/auth/token/create method=POST
...

    This time, instead of using the auto-auth token, the command is using the student token generated upon successful login. The resulting token was successfully cached.

    NOTE: The agent will automatically renew the token to keep it valid:

    [DEBUG] cache.leasecache: renewal received; updating cache: path=/v1/auth/token/create

  6. Re-send the token create request.

    $ vault token create -ttl=12m

Key                  Value
---                  -----
token                s.e1h5wPbZ7i6FLCjPDqpsfDBG
token_accessor       y8K7xJr15fFjSowWW1t3VERT
...

    The same token should be returned, and in the agent log, find the following:

    [INFO]  cache: received request: path=/v1/auth/token/create method=POST
[DEBUG] cache.leasecache: returning cached response: path=/v1/auth/token/create

API call using cURL

  1. Execute the following command to request an AWS credential.

    $ curl -s $VAULT_AGENT_ADDR/v1/aws/creds/readonly | jq
 {
   "request_id": "147861fb-6816-686d-2ad9-01cc15eebbdf",
   "lease_id": "aws/creds/readonly/zMtvdZ2uHHlWVx6PyHsk6dp8",
   ...
   "data": {
     "access_key": "AKIAJMRXXXXXXXXMAGIA",
     "secret_key": "swNXag6aVaultHappyYayAaivQ1iwMBaKCvuDoI",
     "security_token": null
   },
   ...
 }

  2. See the logs in the terminal where Vault Agent is running.

    ...
[INFO]  cache: received request: path=/v1/aws/creds/readonly method=GET
[DEBUG] cache: using auto auth token: path=/v1/aws/creds/readonly method=GET
[DEBUG] cache.leasecache: forwarding request: path=/v1/aws/creds/readonly method=GET
[INFO]  cache.apiproxy: forwarding request: path=/v1/aws/creds/readonly method=GET
[DEBUG] cache.leasecache: processing lease response: path=/v1/aws/creds/readonly method=GET
[DEBUG] cache.leasecache: storing response into the cache: path=/v1/aws/creds/readonly method=GET
[DEBUG] cache.leasecache: initiating renewal: path=/v1/aws/creds/readonly method=GET
[DEBUG] cache.leasecache: secret renewed: path=/v1/aws/creds/readonly

    The log indicates that auto-auth token was used to connect Vault. The received request was forwarded to the Vault server, and the returned response was cached (find an entry storing response into the cache in the log).

  3. Re-run the request to study agent behavior:

    $ curl -s $VAULT_AGENT_ADDR/v1/aws/creds/readonly | jq
 {
   "request_id": "147861fb-6816-686d-2ad9-01cc15eebbdf",
   "lease_id": "aws/creds/readonly/zMtvdZ2uHHlWVx6PyHsk6dp8",
   ...
   "data": {
     "access_key": "AKIAJMRXXXXXXXXMAGIA",
     "secret_key": "swNXag6aVaultHappyYayAaivQ1iwMBaKCvuDoI",
     ...
 }

    The agent log indicates:

    [INFO]  cache: received request: path=/v1/aws/creds/readonly method=GET
[DEBUG] cache: using auto auth token: path=/v1/aws/creds/readonly method=GET
[DEBUG] cache.leasecache: returning cached response: path=/v1/aws/creds/readonly

  4. Log in as a user, student via agent:

    # Login with username 'student' and password is "pAssw0rd"
$ curl --request POST --data '{"password": "pAssw0rd"}' \
       $VAULT_AGENT_ADDR/v1/auth/userpass/login/student | jq
{
  ...
  "auth": {
    "client_token": "s.3vfZXvNcgiIGdJM5gqdSkOlo",
    "accessor": "uPJQluNRr0IteCDElkASBPoa",
    "policies": [
      "default",
      "myapp"
    ],
    ...
}

# Store the acquired token in VAULT_TOKEN environment variable
$ export VAULT_TOKEN="s.3vfZXvNcgiIGdJM5gqdSkOlo"

    Examine the agent log in the other terminal:

    ...
[INFO]  cache: received request: path=/v1/auth/userpass/login/student method=POST
[DEBUG] cache: using auto auth token: path=/v1/auth/userpass/login/student method=POST
[DEBUG] cache.leasecache: forwarding request: path=/v1/auth/userpass/login/student method=POST
[INFO]  cache.apiproxy: forwarding request: path=/v1/auth/userpass/login/student method=POST
[DEBUG] cache.leasecache: processing auth response: path=/v1/auth/userpass/login/student method=POST
[DEBUG] cache.leasecache: storing response into the cache: path=/v1/auth/userpass/login/student method=POST
[DEBUG] cache.leasecache: initiating renewal: path=/v1/auth/userpass/login/student method=POST
[DEBUG] cache.leasecache: secret renewed: path=/v1/auth/userpass/login/student
...

    The log should indicate that the acquired token for user student is successfully cached ([DEBUG] cache.leasecache: storing response into the cache). This is because the login request was sent through the agent.

  5. Using the student token, create a new token with TTL set to 12 minutes:

    $ curl --header "X-Vault-Token: $VAULT_TOKEN" \
       --data '{"ttl": "12m"}' $VAULT_AGENT_ADDR/v1/auth/token/create | jq
 {
   ...
   "auth": {
     "client_token": "s.AkdB7Rgr0t8mIexWEGHPsNfn",
     "accessor": "XAUhhmWj4tgHTpkijamX11iE",
     "policies": [
       "default",
       "myapp"
     ],
     ...
     "token_type": "service"
   }
 }

    Examine the agent log in the other terminal:

    ...
[INFO]  cache: received request: path=/v1/auth/token/create method=POST
[DEBUG] cache.leasecache: forwarding request: path=/v1/auth/token/create method=POST
[INFO]  cache.apiproxy: forwarding request: path=/v1/auth/token/create method=POST
[DEBUG] cache.leasecache: processing auth response: path=/v1/auth/token/create method=POST
[DEBUG] cache.leasecache: setting parent context: path=/v1/auth/token/create method=POST
[DEBUG] cache.leasecache: storing response into the cache: path=/v1/auth/token/create method=POST
...

    This time, instead of using the auto-auth token, you provided student token in the request header (X-Vault-Token). The resulting token was successfully cached.

    NOTE: After a few minutes, the agent will automatically renew the token to keep it valid:

    [DEBUG] cache.leasecache: renewal received; updating cache: path=/v1/auth/token/create

  6. Re-send the token create request:

    $ curl --header "X-Vault-Token: $VAULT_TOKEN" \
       --data '{"ttl": "12m"}' $VAULT_AGENT_ADDR/v1/auth/token/create | jq
 {
   ...
   "auth": {
     "client_token": "s.AkdB7Rgr0t8mIexWEGHPsNfn",
     ...
   }
 }

    The same token should be returned, and in the agent log, find the following:

    [INFO]  cache: received request: path=/v1/auth/token/create method=POST
[DEBUG] cache.leasecache: returning cached response: path=/v1/auth/token/create

Step 5: Cache Evictions

The eviction of cache will occur when the agent fails to renew leases or tokens. This can happen when the cached lease/token hits it's maximum TTL or if the renewal results in an error. Agent also does some best-effort cache evictions by observing specific request types and response codes.

While agent observes requests and evicts cached entries automatically, you can trigger a cache eviction by invoking the /agent/v1/cache-clear endpoint.

  1. If you need to manually evict a stale lease, invoke the /agent/v1/cache-clear endpoint with lease ID for which you wish to evict from the cache.

    Example:

    $ curl --request POST \
       --data '{ "type": "lease", "value": "aws/creds/readonly/8DLXo8nsgcIpx7iQPpCvWtlc" }' \
       $VAULT_AGENT_ADDR/agent/v1/cache-clear

    The agent log should show the following:

    [DEBUG] cache.leasecache: received cache-clear request: type=lease namespace= value=aws/creds/readonly/8DLXo8nsgcIpx7iQPpCvWtlc
[DEBUG] cache.leasecache: canceling context of index attached to accessor
[DEBUG] cache.leasecache: successfully cleared matching cache entries
[DEBUG] cache.leasecache: context cancelled; stopping renewer: path=/v1/aws/creds/readonly
[DEBUG] cache.leasecache: evicting index from cache: id=dd0f9e775... path=/v1/aws/creds/readonly method=GET

  2. Let's see what happens when you revoke a token.

    # Revoke a token via CLI
$ vault token revoke s.AkdB7Rgr0t8mIexWEGHPsNfn

# Or, revoke a token via API
$ curl --request POST \
       --data '{"token": "s.AkdB7Rgr0t8mIexWEGHPsNfn"}' \
       $VAULT_AGENT_ADDR/v1/auth/token/revoke

    Examine the agent log:

    ...
[INFO]  cache: received request: path=/v1/auth/token/revoke method=POST
[DEBUG] cache: using auto auth token: path=/v1/auth/token/revoke method=POST
[DEBUG] cache.leasecache: forwarding request: path=/v1/auth/token/revoke method=POST
[INFO]  cache.apiproxy: forwarding request: path=/v1/auth/token/revoke method=POST
[DEBUG] cache.leasecache: cancelling context of index attached to token
[DEBUG] cache.leasecache: successfully cleared matching cache entries
[DEBUG] cache.leasecache: triggered caching eviction from revocation request
[DEBUG] cache.leasecache: context cancelled; stopping renewer: path=/v1/auth/token/create
[DEBUG] cache.leasecache: evicting index from cache: id=b5715bdca771174... path=/v1/auth/token/create method=POST

    Similarly, if you revoked a staled AWS lease, the agent will automatically evict the cache.

  3. If a situation requires you to clear all cached tokens and leases (e.g. reset after a number of testing), set the type to all.

    $ curl --request POST --data '{ "type": "all" }' \
       $VAULT_AGENT_ADDR/agent/v1/cache-clear

Step 6: Clean Up

  1. On the server SSH terminal, execute the following command to revoke all leases.

    $ vault lease revoke -prefix aws/creds/readonly

  2. Execute the following commands to destroy cloud resources.

    $ terraform destroy -force

$ rm -rf .terraform terraform.tfstate*

Help and Reference

Previous
Vault Agent with Kubernetes
Keep Learning
Explore Operations & Development Tracks
On This Page