Vault Agent Caching | Vault

Nearly all requests to Vault must be accompanied by a valid token which means that Vault clients must first authenticate with Vault and acquire a token. Secure Introduction of Vault Clients tutorial talked about the approaches to solve this secret zero problem.

Vault Agent with AWS and Vault Agent with Kubernetes tutorials walked through the Vault Agent's Auto-Auth.

Vault Agent

NOTE: This tutorial focuses on the Caching feature of Vault Agent.

»Challenge & Solution

Depending on the location of your Vault clients and its secret access frequency, you may face some scaling or latency challenge. Even with Vault Performance Replication enabled, the pressure on the storage backend increases as the number of token or lease generation requests increase. Vault 1.0 introduced batch tokens as a solution to relieve some pressure on the storage backend. By design, batch tokens do not support the same level of flexibility and features as service tokens. Therefore, if you need a periodic token for example, you would need service tokens.

To increase the availability of tokens and secrets to the clients, Vault Agent introduced the Caching function.

Vault Agent Caching

Vault Agent Caching can cache the tokens and leased secrets proxied through the agent which includes the auto-auth token. This allows for easier access to Vault secrets for edge applications, reduces the I/O burden for basic secrets access for Vault clusters, and allows for secure local access to leased secrets for the life of a valid token.

»Prerequisites

To complete this section of the tutorial, you will need:

AWS account and associated credentials that allow for the creation of resources
Terraform installed

NOTE: An interactive tutorial is also available to demonstrate the basic working of Vault Agent if you do not have an AWS account to perform the steps described in this tutorial. Click the Show Terminal button to start.

»Provision the Cloud Resources

Clone or download the demo assets from the hashicorp/vault-guides GitHub repository to perform the steps described in this tutorial.
Clone the repository:
```
$ git clone https://github.com/hashicorp/vault-guides.git
```
```
$ git clone https://github.com/hashicorp/vault-guides.git
```
Or download the repository:
This repository contains supporting content for all of the Vault learn tutorials. The content specific to this tutorial can be found within a sub-directory.

Set your working directory to where the /identity/vault-agent-caching/terraform-aws was downloaded.

$ cd vault-guides/identity/vault-agent-caching/terraform-aws

$ cd vault-guides/identity/vault-agent-caching/terraform-aws

The directory contains the following files.

$ tree
.
├── aws.tf
├── iam.tf
├── instances.tf
├── kms.tf
├── network.tf
├── outputs.tf
├── security-groups.tf
├── templates
│   ├── userdata-vault-client.tpl
│   └── userdata-vault-server.tpl
├── terraform.tfvars.example
├── variables.tf
└── versions.tf

1 directory, 12 files

$ tree.├── aws.tf├── iam.tf├── instances.tf├── kms.tf├── network.tf├── outputs.tf├── security-groups.tf├── templates│   ├── userdata-vault-client.tpl│   └── userdata-vault-server.tpl├── terraform.tfvars.example├── variables.tf└── versions.tf
1 directory, 12 files

NOTE: The example Terraform in this repository is created for the demo purpose.

Set an AWS_ACCESS_KEY_ID environment variable to hold your AWS access key ID.

$ export AWS_ACCESS_KEY_ID = "<YOUR_AWS_ACCESS_KEY_ID>"

$ export AWS_ACCESS_KEY_ID = "<YOUR_AWS_ACCESS_KEY_ID>"

Set an AWS_SECRET_ACCESS_KEY environment variable to hold your AWS secret access key.

$ export AWS_SECRET_ACCESS_KEY = "<YOUR_AWS_SECRET_ACCESS_KEY>"

$ export AWS_SECRET_ACCESS_KEY = "<YOUR_AWS_SECRET_ACCESS_KEY>"

Create a file named terraform.tfvars and specify the key_name. (You can copy the terraform.tfvars.example file and rename it as terraform.tfvars.)
Example terraform.tfvars:
```
# EC2 key pair name to access EC2 instances (should already exist) on the AWS region
key_name = "vault-test"
```
```
# EC2 key pair name to access EC2 instances (should already exist) on the AWS regionkey_name = "vault-test"
```
If you don't have an EC2 key pair, follow the AWS documentation to create one.
Perform a terraform init to pull down the necessary provider resources.
```
$ terraform init
```
```
$ terraform init
```

Run terraform apply to provision AWS resources.

$ terraform apply

$ terraform apply

When prompted, enter yes to proceed.

The Terraform output will display the public IP address to SSH into your Vault server and client instances.

...
Apply complete! Resources: 23 added, 0 changed, 0 destroyed.

Outputs:

endpoints =
Vault Server IP (public):  54.219.129.15
Vault Server IP (private): 10.0.101.50

For example:
   ssh -i vault-test.pem ubuntu@54.219.129.15

Vault Client IP (public):  54.183.212.51
Vault Client IP (private): 10.0.101.209

For example:
   ssh -i vault-test.pem ubuntu@54.183.212.51

...Apply complete! Resources: 23 added, 0 changed, 0 destroyed.
Outputs:
endpoints =Vault Server IP (public):  54.219.129.15Vault Server IP (private): 10.0.101.50
For example:   ssh -i vault-test.pem ubuntu@54.219.129.15
Vault Client IP (public):  54.183.212.51Vault Client IP (private): 10.0.101.209
For example:   ssh -i vault-test.pem ubuntu@54.183.212.51

»Configure AWS IAM Auth Method

This step should be performed on the Vault Server instance.

In this step, you will configure Vault to allow AWS IAM authentication from specific IAM roles.

SSH into the Vault Server instance.

Output

When you are prompted, enter "yes" to continue.

$ ssh -i <path_to_key> ubuntu@<public_ip_of_server>
...
Are you sure you want to continue connecting (yes/no)? yes

$ ssh -i <path_to_key> ubuntu@<public_ip_of_server>...Are you sure you want to continue connecting (yes/no)? yes

NOTE: If you received Permissions 0664 for '<key_name>.pem' are too open error, be sure to set the file permission appropriately.

$ chmod 600 <key_name>.pem

$ chmod 600 <key_name>.pem

Run the vault operator init command to initialize the Vault server.
```
$ vault operator init
...
Initial Root Token: s.20JnHBY66EKTj9zyR6SjTMNq

Success! Vault is initialized
...
```
```
$ vault operator init...Initial Root Token: s.20JnHBY66EKTj9zyR6SjTMNq
Success! Vault is initialized...
```
NOTE: The Vault server is configured to auto-unseal with AWS Key Management Service (KMS); therefore, once the server is initialized, it gets automatically unsealed. To learn how to auto-unseal Vault using AWS KMS, refer to the Auto-unseal using AWS KMS tutorial.
Copy the Initial Root Token value.

Log into Vault using the generated initial root token.

$ vault login s.20JnHBY66EKTj9zyR6SjTMNq

$ vault login s.20JnHBY66EKTj9zyR6SjTMNq

Examine and then execute the /home/ubuntu/aws_auth.sh script.
```
$ cat aws_auth.sh
```
```
$ cat aws_auth.sh
```

Execute the script.

$ ./aws_auth.sh

Success! Enabled the kv secrets engine at: kv/
Success! Uploaded policy: myapp
Success! Enabled aws auth method at: aws/
Success! Data written to: auth/aws/config/client
Success! Data written to: auth/aws/role/app-role
Success! Enabled userpass auth method at: userpass/
Success! Data written to: auth/userpass/users/student

$ ./aws_auth.sh
Success! Enabled the kv secrets engine at: kv/Success! Uploaded policy: myappSuccess! Enabled aws auth method at: aws/Success! Data written to: auth/aws/config/clientSuccess! Data written to: auth/aws/role/app-roleSuccess! Enabled userpass auth method at: userpass/Success! Data written to: auth/userpass/users/student

This script enables key/value v1 secrets engine at kv, create myapp policy, enables aws auth method and create a role named app-role. Also, userpass is enabled and created a user (student) with myapp policy attached.

Check the myapp policy.

$ vault policy read myapp

path "kv/*" {
  capabilities = ["create", "read", "update", "delete"]
}
path "aws/creds/*" {
  capabilities = ["read", "update"]
}
path "sys/leases/*" {
  capabilities = ["create", "update"]
}
path "auth/token/*" {
  capabilities = ["create", "update"]
}

$ vault policy read myapp
path "kv/*" {  capabilities = ["create", "read", "update", "delete"]}path "aws/creds/*" {  capabilities = ["read", "update"]}path "sys/leases/*" {  capabilities = ["create", "update"]}path "auth/token/*" {  capabilities = ["create", "update"]}

Examine and then execute the /home/ubuntu/aws_secrets.sh script.
```
$ cat aws_secrets.sh
```
```
$ cat aws_secrets.sh
```

Execute the script.

$ ./aws_secrets.sh
Success! Enabled the aws secrets engine at: aws/
Success! Data written to: aws/config/root
Success! Data written to: aws/config/lease
Success! Data written to: aws/roles/readonly

$ ./aws_secrets.shSuccess! Enabled the aws secrets engine at: aws/Success! Data written to: aws/config/rootSuccess! Data written to: aws/config/leaseSuccess! Data written to: aws/roles/readonly

This script enables aws secrets engine, configures it and set the generated secret's lease to 1 hour and lease max to 24 hours. It also configures readonly role which is mapped to is mapped to arn:aws:iam::aws:policy/AmazonS3ReadOnlyAccess IAM policy.

»Start Vault Agent

This step should be performed on the Vault Client instance.

Now, SSH into the Vault Client instance.

Output

When you are prompted, enter "yes" to continue.

$ ssh -i <path_to_key> ubuntu@<public_ip_of_client>
...
Are you sure you want to continue connecting (yes/no)? yes

$ ssh -i <path_to_key> ubuntu@<public_ip_of_client>...Are you sure you want to continue connecting (yes/no)? yes

Explore the Vault Agent configuration file, /home/ubuntu/vault-agent.hcl

$ cat vault-agent.hcl

exit_after_auth = false
pid_file = "./pidfile"

auto_auth {
   method "aws" {
       mount_path = "auth/aws"
       config = {
           type = "iam"
           role = "app-role"
       }
   }

   sink "file" {
       config = {
           path = "/home/ubuntu/vault-token-via-agent"
       }
   }
}

cache {
   use_auto_auth_token = true
}

listener "tcp" {
   address = "127.0.0.1:8200"
   tls_disable = true
}

vault {
   address = "http://<vault-server-host>:8200"
}

$ cat vault-agent.hcl
exit_after_auth = falsepid_file = "./pidfile"
auto_auth {   method "aws" {       mount_path = "auth/aws"       config = {           type = "iam"           role = "app-role"       }   }
   sink "file" {       config = {           path = "/home/ubuntu/vault-token-via-agent"       }   }}
cache {   use_auto_auth_token = true}
listener "tcp" {   address = "127.0.0.1:8200"   tls_disable = true}
vault {   address = "http://<vault-server-host>:8200"}

Also, read the Server Side Request Forgery (SSRF) Protection section.

Notice the cache block sets the use_auto_auth_token parameter to true. Since the auto-auth sink is set to /home/ubuntu/vault-token-via-agent, in absence of client token, Vault Agent will use the auto-auth token read from vault-token-via-agent. The vault block defines the Vault server location (e.g. http://10.0.101.209:8200).

Execute the following command to start the Vault Agent with log level set to debug.

$ vault agent -config=/home/ubuntu/vault-agent.hcl -log-level=debug

==> Vault server started! Log data will stream in below:

==> Vault agent configuration:

           Api Address 1: http://127.0.0.1:8200
                     Cgo: disabled
               Log Level: debug
                 Version: Vault v1.3.0
             Version Sha: 30f07c76e1ea0551f28f5b8f7537f0de49d27f80

2019-03-06T03:12:47.616Z [INFO]  sink.file: creating file sink
2019-03-06T03:12:47.616Z [INFO]  sink.file: file sink configured: path=/home/ubuntu/vault-token-via-agent
2019-03-06T03:12:47.617Z [DEBUG] cache: auto-auth token is allowed to be used; configuring inmem sink
2019-03-06T03:12:47.617Z [INFO]  cache: starting listener: addr=127.0.0.1:8200
2019-03-06T03:12:47.618Z [INFO]  auth.handler: starting auth handler
2019-03-06T03:12:47.618Z [INFO]  auth.handler: authenticating
2019-03-06T03:12:47.618Z [INFO]  sink.server: starting sink server
2019-03-06T03:12:47.883Z [INFO]  auth.handler: authentication successful, sending token to sinks
2019-03-06T03:12:47.883Z [INFO]  auth.handler: starting renewal process
2019-03-06T03:12:47.883Z [INFO]  sink.file: token written: path=/home/ubuntu/vault-token-via-agent
2019-03-06T03:12:47.883Z [DEBUG] cache.leasecache: storing auto-auth token into the cache
2019-03-06T03:12:47.887Z [INFO]  auth.handler: renewed auth token

$ vault agent -config=/home/ubuntu/vault-agent.hcl -log-level=debug
==> Vault server started! Log data will stream in below:
==> Vault agent configuration:
           Api Address 1: http://127.0.0.1:8200                     Cgo: disabled               Log Level: debug                 Version: Vault v1.3.0             Version Sha: 30f07c76e1ea0551f28f5b8f7537f0de49d27f80
2019-03-06T03:12:47.616Z [INFO]  sink.file: creating file sink2019-03-06T03:12:47.616Z [INFO]  sink.file: file sink configured: path=/home/ubuntu/vault-token-via-agent2019-03-06T03:12:47.617Z [DEBUG] cache: auto-auth token is allowed to be used; configuring inmem sink2019-03-06T03:12:47.617Z [INFO]  cache: starting listener: addr=127.0.0.1:82002019-03-06T03:12:47.618Z [INFO]  auth.handler: starting auth handler2019-03-06T03:12:47.618Z [INFO]  auth.handler: authenticating2019-03-06T03:12:47.618Z [INFO]  sink.server: starting sink server2019-03-06T03:12:47.883Z [INFO]  auth.handler: authentication successful, sending token to sinks2019-03-06T03:12:47.883Z [INFO]  auth.handler: starting renewal process2019-03-06T03:12:47.883Z [INFO]  sink.file: token written: path=/home/ubuntu/vault-token-via-agent2019-03-06T03:12:47.883Z [DEBUG] cache.leasecache: storing auto-auth token into the cache2019-03-06T03:12:47.887Z [INFO]  auth.handler: renewed auth token

Open a second SSH terminal into the client machine.
Verify the client token stored in /home/ubuntu/vault-token-via-agent.
```
$ more vault-token-via-agent
```
```
$ more vault-token-via-agent
```

Check the details about the token (e.g. attached policies, TTL, etc.).

$ VAULT_TOKEN="$(cat vault-token-via-agent)" vault token lookup

$ VAULT_TOKEN="$(cat vault-token-via-agent)" vault token lookup

Set VAULT_AGENT_ADDR environment variable.

$ export VAULT_AGENT_ADDR="http://127.0.0.1:8200"

$ export VAULT_AGENT_ADDR="http://127.0.0.1:8200"

»Request dynamic secrets and tokens via agent

Execute the following command to request an AWS credential.

$ vault read aws/creds/readonly

Key                Value
---                -----
lease_id           aws/creds/readonly/oWBfLT2nIhXq9ZgWaC2e1fHy
lease_duration     1h
lease_renewable    true
access_key         AKIAJMRXXXXXXXXMAGIA
secret_key         swNXag6aVaultHappyYayAaivQ1iwMBaKCvuDoI
security_token     <nil>

$ vault read aws/creds/readonly
Key                Value---                -----lease_id           aws/creds/readonly/oWBfLT2nIhXq9ZgWaC2e1fHylease_duration     1hlease_renewable    trueaccess_key         AKIAJMRXXXXXXXXMAGIAsecret_key         swNXag6aVaultHappyYayAaivQ1iwMBaKCvuDoIsecurity_token     <nil>

See the logs in the terminal where Vault Agent is running.

...
[INFO]  cache: received request: path=/v1/aws/creds/readonly method=GET
[DEBUG] cache: using auto auth token: path=/v1/aws/creds/readonly method=GET
[DEBUG] cache.leasecache: forwarding request: path=/v1/aws/creds/readonly method=GET
[INFO]  cache.apiproxy: forwarding request: path=/v1/aws/creds/readonly method=GET
[DEBUG] cache.leasecache: processing lease response: path=/v1/aws/creds/readonly method=GET
[DEBUG] cache.leasecache: storing response into the cache: path=/v1/aws/creds/readonly method=GET
[DEBUG] cache.leasecache: initiating renewal: path=/v1/aws/creds/readonly method=GET
[DEBUG] cache.leasecache: secret renewed: path=/v1/aws/creds/readonly

...[INFO]  cache: received request: path=/v1/aws/creds/readonly method=GET[DEBUG] cache: using auto auth token: path=/v1/aws/creds/readonly method=GET[DEBUG] cache.leasecache: forwarding request: path=/v1/aws/creds/readonly method=GET[INFO]  cache.apiproxy: forwarding request: path=/v1/aws/creds/readonly method=GET[DEBUG] cache.leasecache: processing lease response: path=/v1/aws/creds/readonly method=GET[DEBUG] cache.leasecache: storing response into the cache: path=/v1/aws/creds/readonly method=GET[DEBUG] cache.leasecache: initiating renewal: path=/v1/aws/creds/readonly method=GET[DEBUG] cache.leasecache: secret renewed: path=/v1/aws/creds/readonly

The log indicates that auto-auth token was used to connect Vault. The received request was forwarded to the Vault server, and the returned response was cached (find an entry storing response into the cache in the log).

Benefit of Vault Agent: Vault Agent will manage the lifecycle of cached tokens and leases automatically so that the clients do not need to implement a logic to renew the tokens and leases.

Re-run the vault command to study the agent behavior.

$ vault read aws/creds/readonly

Key                Value
---                -----
lease_id           aws/creds/readonly/oWBfLT2nIhXq9ZgWaC2e1fHy
lease_duration     1h
lease_renewable    true
access_key         AKIAJMRXXXXXXXXMAGIA
secret_key         swNXag6aVaultHappyYayAaivQ1iwMBaKCvuDoI
security_token     <nil>

$ vault read aws/creds/readonly
Key                Value---                -----lease_id           aws/creds/readonly/oWBfLT2nIhXq9ZgWaC2e1fHylease_duration     1hlease_renewable    trueaccess_key         AKIAJMRXXXXXXXXMAGIAsecret_key         swNXag6aVaultHappyYayAaivQ1iwMBaKCvuDoIsecurity_token     <nil>

The agent log indicates:

[INFO]  cache: received request: path=/v1/aws/creds/readonly method=GET
[DEBUG] cache: using auto auth token: path=/v1/aws/creds/readonly method=GET
[DEBUG] cache.leasecache: returning cached response: path=/v1/aws/creds/readonly

[INFO]  cache: received request: path=/v1/aws/creds/readonly method=GET[DEBUG] cache: using auto auth token: path=/v1/aws/creds/readonly method=GET[DEBUG] cache.leasecache: returning cached response: path=/v1/aws/creds/readonly

This returns the same AWS credentials. Notice that the request_id and lease_id match to the ones you received the first time.

# Login with username 'student' and password is "pAssw0rd"
$ vault login -method=userpass username="student" password="pAssw0rd"

Success! You are now authenticated. The token information displayed below
is already stored in the token helper. You do NOT need to run "vault login"
again. Future Vault requests will automatically use this token.

Key                    Value
---                    -----
token                  s.AKsRLji9mPq4xNZP1UIJggk6
token_accessor         lU69kHGAPtYMoQndeCSoYuhs
token_duration         48h
token_renewable        true
token_policies         ["default" "myapp"]
...

# Store the acquired token in VAULT_TOKEN environment variable
$ export VAULT_TOKEN="s.AKsRLji9mPq4xNZP1UIJggk6"

# Login with username 'student' and password is "pAssw0rd"$ vault login -method=userpass username="student" password="pAssw0rd"
Success! You are now authenticated. The token information displayed belowis already stored in the token helper. You do NOT need to run "vault login"again. Future Vault requests will automatically use this token.
Key                    Value---                    -----token                  s.AKsRLji9mPq4xNZP1UIJggk6token_accessor         lU69kHGAPtYMoQndeCSoYuhstoken_duration         48htoken_renewable        truetoken_policies         ["default" "myapp"]...
# Store the acquired token in VAULT_TOKEN environment variable$ export VAULT_TOKEN="s.AKsRLji9mPq4xNZP1UIJggk6"

Examine the agent log in the other terminal.

...
[INFO]  cache: received request: path=/v1/auth/userpass/login/student method=POST
[DEBUG] cache: using auto auth token: path=/v1/auth/userpass/login/student method=POST
[DEBUG] cache.leasecache: forwarding request: path=/v1/auth/userpass/login/student method=POST
[INFO]  cache.apiproxy: forwarding request: path=/v1/auth/userpass/login/student method=POST
[DEBUG] cache.leasecache: processing auth response: path=/v1/auth/userpass/login/student method=POST
[DEBUG] cache.leasecache: storing response into the cache: path=/v1/auth/userpass/login/student method=POST
[DEBUG] cache.leasecache: initiating renewal: path=/v1/auth/userpass/login/student method=POST
[DEBUG] cache.leasecache: secret renewed: path=/v1/auth/userpass/login/student
...

...[INFO]  cache: received request: path=/v1/auth/userpass/login/student method=POST[DEBUG] cache: using auto auth token: path=/v1/auth/userpass/login/student method=POST[DEBUG] cache.leasecache: forwarding request: path=/v1/auth/userpass/login/student method=POST[INFO]  cache.apiproxy: forwarding request: path=/v1/auth/userpass/login/student method=POST[DEBUG] cache.leasecache: processing auth response: path=/v1/auth/userpass/login/student method=POST[DEBUG] cache.leasecache: storing response into the cache: path=/v1/auth/userpass/login/student method=POST[DEBUG] cache.leasecache: initiating renewal: path=/v1/auth/userpass/login/student method=POST[DEBUG] cache.leasecache: secret renewed: path=/v1/auth/userpass/login/student...

The log should indicate that the acquired token for user student is successfully cached ([DEBUG] cache.leasecache: storing response into the cache). This is because the login request was sent through the agent.

Create a new token with TTL set to 12 minutes.

$ vault token create -ttl=12m

Key                  Value
---                  -----
token                s.e1h5wPbZ7i6FLCjPDqpsfDBG
token_accessor       y8K7xJr15fFjSowWW1t3VERT
token_duration       12m
...

$ vault token create -ttl=12m
Key                  Value---                  -----token                s.e1h5wPbZ7i6FLCjPDqpsfDBGtoken_accessor       y8K7xJr15fFjSowWW1t3VERTtoken_duration       12m...

Examine the agent log in the other terminal.

...
[INFO]  cache: received request: path=/v1/auth/token/create method=POST
[DEBUG] cache.leasecache: forwarding request: path=/v1/auth/token/create method=POST
[INFO]  cache.apiproxy: forwarding request: path=/v1/auth/token/create method=POST
[DEBUG] cache.leasecache: processing auth response: path=/v1/auth/token/create method=POST
[DEBUG] cache.leasecache: setting parent context: path=/v1/auth/token/create method=POST
[DEBUG] cache.leasecache: storing response into the cache: path=/v1/auth/token/create method=POST
...

...[INFO]  cache: received request: path=/v1/auth/token/create method=POST[DEBUG] cache.leasecache: forwarding request: path=/v1/auth/token/create method=POST[INFO]  cache.apiproxy: forwarding request: path=/v1/auth/token/create method=POST[DEBUG] cache.leasecache: processing auth response: path=/v1/auth/token/create method=POST[DEBUG] cache.leasecache: setting parent context: path=/v1/auth/token/create method=POST[DEBUG] cache.leasecache: storing response into the cache: path=/v1/auth/token/create method=POST...

This time, instead of using the auto-auth token, the command is using the student token generated upon successful login. The resulting token was successfully cached.

NOTE: The agent will automatically renew the token to keep it valid:

[DEBUG] cache.leasecache: renewal received; updating cache: path=/v1/auth/token/create

[DEBUG] cache.leasecache: renewal received; updating cache: path=/v1/auth/token/create

Re-send the token create request.

$ vault token create -ttl=12m

Key                  Value
---                  -----
token                s.e1h5wPbZ7i6FLCjPDqpsfDBG
token_accessor       y8K7xJr15fFjSowWW1t3VERT
...

$ vault token create -ttl=12m
Key                  Value---                  -----token                s.e1h5wPbZ7i6FLCjPDqpsfDBGtoken_accessor       y8K7xJr15fFjSowWW1t3VERT...

The same token should be returned, and in the agent log, find the following:

[INFO]  cache: received request: path=/v1/auth/token/create method=POST
[DEBUG] cache.leasecache: returning cached response: path=/v1/auth/token/create

[INFO]  cache: received request: path=/v1/auth/token/create method=POST[DEBUG] cache.leasecache: returning cached response: path=/v1/auth/token/create

»Cache Evictions

The eviction of cache will occur when the agent fails to renew leases or tokens. This can happen when the cached lease/token hits it's maximum TTL or if the renewal results in an error. Agent also does some best-effort cache evictions by observing specific request types and response codes.

While agent observes requests and evicts cached entries automatically, you can trigger a cache eviction by invoking the /agent/v1/cache-clear endpoint.

If you need to manually evict a stale lease, invoke the /agent/v1/cache-clear endpoint with lease ID for which you wish to evict from the cache.

Example:

$ curl --request POST \
       --data '{ "type": "lease", "value": "aws/creds/readonly/8DLXo8nsgcIpx7iQPpCvWtlc" }' \
       $VAULT_AGENT_ADDR/agent/v1/cache-clear

$ curl --request POST \       --data '{ "type": "lease", "value": "aws/creds/readonly/8DLXo8nsgcIpx7iQPpCvWtlc" }' \       $VAULT_AGENT_ADDR/agent/v1/cache-clear

The agent log should show the following:

[DEBUG] cache.leasecache: received cache-clear request: type=lease namespace= value=aws/creds/readonly/8DLXo8nsgcIpx7iQPpCvWtlc
[DEBUG] cache.leasecache: canceling context of index attached to accessor
[DEBUG] cache.leasecache: successfully cleared matching cache entries
[DEBUG] cache.leasecache: context cancelled; stopping renewer: path=/v1/aws/creds/readonly
[DEBUG] cache.leasecache: evicting index from cache: id=dd0f9e775... path=/v1/aws/creds/readonly method=GET

[DEBUG] cache.leasecache: received cache-clear request: type=lease namespace= value=aws/creds/readonly/8DLXo8nsgcIpx7iQPpCvWtlc[DEBUG] cache.leasecache: canceling context of index attached to accessor[DEBUG] cache.leasecache: successfully cleared matching cache entries[DEBUG] cache.leasecache: context cancelled; stopping renewer: path=/v1/aws/creds/readonly[DEBUG] cache.leasecache: evicting index from cache: id=dd0f9e775... path=/v1/aws/creds/readonly method=GET

Let's see what happens when you revoke a token.

$ vault token revoke s.AkdB7Rgr0t8mIexWEGHPsNfn

$ vault token revoke s.AkdB7Rgr0t8mIexWEGHPsNfn

Or, revoke a token via API.

$ curl --request POST \
       --data '{"token": "s.AkdB7Rgr0t8mIexWEGHPsNfn"}' \
       $VAULT_AGENT_ADDR/v1/auth/token/revoke

$ curl --request POST \       --data '{"token": "s.AkdB7Rgr0t8mIexWEGHPsNfn"}' \       $VAULT_AGENT_ADDR/v1/auth/token/revoke

Examine the agent log:

...
[INFO]  cache: received request: path=/v1/auth/token/revoke method=POST
[DEBUG] cache: using auto auth token: path=/v1/auth/token/revoke method=POST
[DEBUG] cache.leasecache: forwarding request: path=/v1/auth/token/revoke method=POST
[INFO]  cache.apiproxy: forwarding request: path=/v1/auth/token/revoke method=POST
[DEBUG] cache.leasecache: cancelling context of index attached to token
[DEBUG] cache.leasecache: successfully cleared matching cache entries
[DEBUG] cache.leasecache: triggered caching eviction from revocation request
[DEBUG] cache.leasecache: context cancelled; stopping renewer: path=/v1/auth/token/create
[DEBUG] cache.leasecache: evicting index from cache: id=b5715bdca771174... path=/v1/auth/token/create method=POST

...[INFO]  cache: received request: path=/v1/auth/token/revoke method=POST[DEBUG] cache: using auto auth token: path=/v1/auth/token/revoke method=POST[DEBUG] cache.leasecache: forwarding request: path=/v1/auth/token/revoke method=POST[INFO]  cache.apiproxy: forwarding request: path=/v1/auth/token/revoke method=POST[DEBUG] cache.leasecache: cancelling context of index attached to token[DEBUG] cache.leasecache: successfully cleared matching cache entries[DEBUG] cache.leasecache: triggered caching eviction from revocation request[DEBUG] cache.leasecache: context cancelled; stopping renewer: path=/v1/auth/token/create[DEBUG] cache.leasecache: evicting index from cache: id=b5715bdca771174... path=/v1/auth/token/create method=POST

In the agent log, notice the message, "triggered caching eviction from revocation request". When a token revocation request is made via the agent, the agent evicts the cached entries associated with the revoked token.

Similarly, if you revoked a staled AWS lease, the agent will automatically evict the cache.

If a situation requires you to clear all cached tokens and leases (e.g. reset after a number of testing), set the type to all.

$ curl --request POST --data '{ "type": "all" }' \
       $VAULT_AGENT_ADDR/agent/v1/cache-clear

$ curl --request POST --data '{ "type": "all" }' \       $VAULT_AGENT_ADDR/agent/v1/cache-clear

»Clean Up

On the server SSH terminal, execute the following command to revoke all leases.

$ vault lease revoke -prefix aws/creds/readonly

$ vault lease revoke -prefix aws/creds/readonly

Execute the following commands to destroy cloud resources.
```
$ terraform destroy -force
```
```
$ terraform destroy -force
```

Delete state files.

$ rm -rf .terraform terraform.tfstate*

$ rm -rf .terraform terraform.tfstate*

»Server Side Request Forgery (SSRF) Protection

NOTE: Vault 1.3 introduced Server Side Request Forgery (SSRF) protection for Vault Agent. To leverage this feature, download Vault 1.3 or later.

Add require_request_header = true in the Agent's listener configuration stanza to enable the SSRF protection.

Example:

exit_after_auth = false
pid_file = "./pidfile"

auto_auth {
   method "aws" {
       mount_path = "auth/aws"
       config = {
           type = "iam"
           role = "app-role"
       }
   }

   sink "file" {
       config = {
           path = "/home/ubuntu/vault-token-via-agent"
       }
   }
}

cache {
   use_auto_auth_token = true
}

listener "tcp" {
   address = "127.0.0.1:8200"
   require_request_header = true
}

vault {
   address = "http://<vault-server-host>:8200"
}

exit_after_auth = falsepid_file = "./pidfile"
auto_auth {   method "aws" {       mount_path = "auth/aws"       config = {           type = "iam"           role = "app-role"       }   }
   sink "file" {       config = {           path = "/home/ubuntu/vault-token-via-agent"       }   }}
cache {   use_auto_auth_token = true}
listener "tcp" {   address = "127.0.0.1:8200"   require_request_header = true}
vault {   address = "http://<vault-server-host>:8200"}

Notice the listener block.

When the require_request_header is set to true, the Agent listener will reject all requests that do NOT have the proper X-Vault-Request: true header.

For example, the command to request AWS credentials you saw in the request dynamic secrets and tokens via agent step must look like:

$ curl -header "X-Vault-Request: true" \
       $VAULT_AGENT_ADDR/v1/aws/creds/readonly

$ curl -header "X-Vault-Request: true" \       $VAULT_AGENT_ADDR/v1/aws/creds/readonly

With absence of the X-Vault-Request header, the Agent will throw missing 'X-Vault-Request' header error, and the request will not be propagated to the Vault server.

Infrastructure

Security

Networking

Applications