HashiCorp Learn
Infrastructure
  • TerraformTerraformLearn terraformDocs
  • PackerPackerLearn packerDocs
  • VagrantVagrantLearn vagrantDocs
Security
  • VaultVaultLearn vaultDocs
  • BoundaryBoundaryLearn boundaryDocs
Networking
  • ConsulConsulLearn consulDocs
Applications
  • NomadNomadLearn nomadDocs
  • WaypointWaypointLearn waypointDocs
  • HashiCorp Cloud Platform (HCP) LogoHashiCorp Cloud Platform (HCP)HashiCorp Cloud Platform (HCP)Docs
Type '/' to Search
Loading account...
  • Bookmarks
  • Manage Account
  • Overview
  • Challenge & Solution
  • Prerequisites
  • Step 1: Provision the Cloud Resources
  • Step 2: Configure AWS IAM Auth Method
  • Step 3: Start Vault Agent
  • Step 4: Request dynamic secrets and tokens via agent
  • Step 5: Cache Evictions
  • Step 6: Clean Up
  • Server Side Request Forgery (SSRF) Protection
  • Help and Reference
DocsForum
Back to vault
App IntegrationView Collection
    Secure Introduction of Vault ClientsVault Agent with AWSVault Agent with KubernetesVault Agent CachingDirect Application IntegrationVault Agent TemplatesAppRole With Terraform & ChefJava Application DemoTransit Secrets Re-wrappingUsing HashiCorp Vault C# Client with .NET CoreUsing HashiCorp Vault Agent with .NET CoreBuild Your Own PluginsVault GitHub ActionsVault AWS Lambda Extension

Vault Agent Caching

  • 17 min
  • Products Usedvaultterraform
  • This tutorial also appears in: Interactive.

Nearly all requests to Vault must be accompanied by a valid token which means that Vault clients must first authenticate with Vault and acquire a token. Secure Introduction of Vault Clients tutorial talked about the approaches to solve this secret zero problem.

Vault Agent with AWS and Vault Agent with Kubernetes tutorials walked through the Vault Agent's Auto-Auth.

Vault Agent

NOTE: This tutorial focuses on the Caching feature of Vault Agent.

»Challenge & Solution

Depending on the location of your Vault clients and its secret access frequency, you may face some scaling or latency challenge. Even with Vault Performance Replication enabled, the pressure on the storage backend increases as the number of token or lease generation requests increase. Vault 1.0 introduced batch tokens as a solution to relieve some pressure on the storage backend. By design, batch tokens do not support the same level of flexibility and features as service tokens. Therefore, if you need a periodic token for example, you would need service tokens.

To increase the availability of tokens and secrets to the clients, Vault Agent introduced the Caching function.

Vault Agent Caching

Vault Agent Caching can cache the tokens and leased secrets proxied through the agent which includes the auto-auth token. This allows for easier access to Vault secrets for edge applications, reduces the I/O burden for basic secrets access for Vault clusters, and allows for secure local access to leased secrets for the life of a valid token.

»Prerequisites

To complete this section of the tutorial, you will need:

  • An AWS account and associated credentials that allow for the creation of resources
  • Vault version 1.1 or later

NOTE: An interactive tutorial is also available to demonstrate the basic working of Vault Agent if you do not have an AWS account to perform the steps described in this tutorial. Click the Show Terminal button to start.

»Step 1: Provision the Cloud Resources

  1. Clone or download the demo assets from the hashicorp/vault-guides GitHub repository to perform the steps described in this tutorial.

    Clone the repository:

    $ git clone https://github.com/hashicorp/vault-guides.git
    

    Or download the repository:

    Download

    This repository contains supporting content for all of the Vault learn tutorials. The content specific to this tutorial can be found within a sub-directory.

  2. Set your working directory to where the /identity/vault-agent-caching/terraform-aws was downloaded.

    $ cd vault-guides/identity/vault-agent-caching/terraform-aws
    

    The directory contains the following files.

    $ tree
    .
    ├── aws.tf
    ├── iam.tf
    ├── instances.tf
    ├── kms.tf
    ├── network.tf
    ├── outputs.tf
    ├── security-groups.tf
    ├── templates
    │   ├── userdata-vault-client.tpl
    │   └── userdata-vault-server.tpl
    ├── terraform.tfvars.example
    ├── variables.tf
    └── versions.tf
    
    1 directory, 12 files
    

    NOTE: The example Terraform in this repository is created for the demo purpose.

  3. Set your AWS credentials as environment variables

    $ export AWS_ACCESS_KEY_ID = ""
    $ export AWS_SECRET_ACCESS_KEY = ""
    
  4. Create a file named terraform.tfvars and specify the key_name. In addition, specify the variable values (variables.tf) which you wish to overwrite its default. (Use the provided terraform.tfvars.example as a base.)

    Example terraform.tfvars:

    # EC2 key pair name to access EC2 instances (should already exist) on the AWS region
    key_name = "vault-test"
    
    # All resources will be tagged with this
    environment_name = "va-demo"
    

    If you don't have an EC2 key pair, follow the AWS documentation to create one.

  5. Perform a terraform init to pull down the necessary provider resources.

    $ terraform init
    
  6. Run terraform apply to provision AWS resources.

    $ terraform apply
    

    When prompted, enter yes to proceed.

    The Terraform output will display the public IP address to SSH into your Vault server and client instances.

    ...
    Apply complete! Resources: 23 added, 0 changed, 0 destroyed.
    
    Outputs:
    
    endpoints =
    Vault Server IP (public):  54.219.129.15
    Vault Server IP (private): 10.0.101.50
    
    For example:
       ssh -i vault-test.pem ubuntu@54.219.129.15
    
    Vault Client IP (public):  54.183.212.51
    Vault Client IP (private): 10.0.101.209
    
    For example:
       ssh -i vault-test.pem ubuntu@54.183.212.51
    

»Step 2: Configure AWS IAM Auth Method

This step should be performed on the Vault Server instance.

In this step, you will configure Vault to allow AWS IAM authentication from specific IAM roles.

  1. SSH into the Vault Server instance.

    Output

    ssh -i  ubuntu@
    ...
    Are you sure you want to continue connecting (yes/no)? yes
    

    When you are prompted, enter "yes" to continue.

  2. Run the vault operator init command to initialize the Vault server.

    $ vault operator init
    ...
    Initial Root Token: s.20JnHBY66EKTj9zyR6SjTMNq
    
    Success! Vault is initialized
    ...
    

    NOTE: The Vault server is configured to auto-unseal with AWS Key Management Service (KMS); therefore, once the server is initialized, it gets automatically unsealed. To learn how to auto-unseal Vault using AWS KMS, refer to the Auto-unseal using AWS KMS tutorial.

  3. Copy the Initial Root Token value.

  4. Log into Vault using the generated initial root token.

    $ vault login s.20JnHBY66EKTj9zyR6SjTMNq
    
  5. Examine and then execute the /home/ubuntu/aws_auth.sh script.

    $ cat aws_auth.sh
    
  6. Execute the script.

    $ ./aws_auth.sh
    
    Success! Enabled the kv secrets engine at: kv/
    Success! Uploaded policy: myapp
    Success! Enabled aws auth method at: aws/
    Success! Data written to: auth/aws/config/client
    Success! Data written to: auth/aws/role/app-role
    Success! Enabled userpass auth method at: userpass/
    Success! Data written to: auth/userpass/users/student
    

    This script enables key/value v1 secrets engine at kv, create myapp policy, enables aws auth method and create a role named app-role. Also, userpass is enabled and created a user (student) with myapp policy attached.

  7. Check the myapp policy.

    $ vault policy read myapp
    
    path "kv/*" {
      capabilities = ["create", "read", "update", "delete"]
    }
    path "aws/creds/*" {
      capabilities = ["read", "update"]
    }
    path "sys/leases/*" {
      capabilities = ["create", "update"]
    }
    path "auth/token/*" {
      capabilities = ["create", "update"]
    }
    
  8. Examine and then execute the /home/ubuntu/aws_secrets.sh script.

    $ cat aws_secrets.sh
    
  9. Execute the script.

    $ ./aws_secrets.sh
    Success! Enabled the aws secrets engine at: aws/
    Success! Data written to: aws/config/root
    Success! Data written to: aws/config/lease
    Success! Data written to: aws/roles/readonly
    

    This script enables aws secrets engine, configures it and set the generated secret's lease to 1 hour and lease max to 24 hours. It also configures readonly role which is mapped to is mapped to arn:aws:iam::aws:policy/AmazonS3ReadOnlyAccess IAM policy.

»Step 3: Start Vault Agent

This step should be performed on the Vault Client instance.

  1. Now, SSH into the Vault Client instance.

    Output

    ssh -i  ubuntu@
    ...
    Are you sure you want to continue connecting (yes/no)? yes
    

    When you are prompted, enter "yes" to continue.

  2. Explore the Vault Agent configuration file, /home/ubuntu/vault-agent.hcl

    $ cat vault-agent.hcl
    
    exit_after_auth = false
    pid_file = "./pidfile"
    
    auto_auth {
       method "aws" {
           mount_path = "auth/aws"
           config = {
               type = "iam"
               role = "app-role"
           }
       }
    
       sink "file" {
           config = {
               path = "/home/ubuntu/vault-token-via-agent"
           }
       }
    }
    
    cache {
       use_auto_auth_token = true
    }
    
    listener "tcp" {
       address = "127.0.0.1:8200"
       tls_disable = true
    }
    
    vault {
       address = "http://<vault-server-host>:8200"
    }
    

    Also, read the Server Side Request Forgery (SSRF) Protection section.

    Notice the cache block sets the use_auto_auth_token parameter to true. Since the auto-auth sink is set to /home/ubuntu/vault-token-via-agent, in absence of client token, Vault Agent will use the auto-auth token read from vault-token-via-agent. The vault block defines the Vault server location (e.g. http://10.0.101.209:8200).

  3. Execute the following command to start the Vault Agent with log level set to debug.

    $ vault agent -config=/home/ubuntu/vault-agent.hcl -log-level=debug
    
    ==> Vault server started! Log data will stream in below:
    
    ==> Vault agent configuration:
    
               Api Address 1: http://127.0.0.1:8200
                         Cgo: disabled
                   Log Level: debug
                     Version: Vault v1.3.0
                 Version Sha: 30f07c76e1ea0551f28f5b8f7537f0de49d27f80
    
    2019-03-06T03:12:47.616Z [INFO]  sink.file: creating file sink
    2019-03-06T03:12:47.616Z [INFO]  sink.file: file sink configured: path=/home/ubuntu/vault-token-via-agent
    2019-03-06T03:12:47.617Z [DEBUG] cache: auto-auth token is allowed to be used; configuring inmem sink
    2019-03-06T03:12:47.617Z [INFO]  cache: starting listener: addr=127.0.0.1:8200
    2019-03-06T03:12:47.618Z [INFO]  auth.handler: starting auth handler
    2019-03-06T03:12:47.618Z [INFO]  auth.handler: authenticating
    2019-03-06T03:12:47.618Z [INFO]  sink.server: starting sink server
    2019-03-06T03:12:47.883Z [INFO]  auth.handler: authentication successful, sending token to sinks
    2019-03-06T03:12:47.883Z [INFO]  auth.handler: starting renewal process
    2019-03-06T03:12:47.883Z [INFO]  sink.file: token written: path=/home/ubuntu/vault-token-via-agent
    2019-03-06T03:12:47.883Z [DEBUG] cache.leasecache: storing auto-auth token into the cache
    2019-03-06T03:12:47.887Z [INFO]  auth.handler: renewed auth token
    
  4. Open a second SSH terminal into the client machine.

  5. Verify the client token stored in /home/ubuntu/vault-token-via-agent.

    $ more vault-token-via-agent
    
  6. Check the details about the token (e.g. attached policies, TTL, etc.).

    $ VAULT_TOKEN="$(cat vault-token-via-agent)" vault token lookup
    
  7. Set VAULT_AGENT_ADDR environment variable.

    $ export VAULT_AGENT_ADDR="http://127.0.0.1:8200"
    

»Step 4: Request dynamic secrets and tokens via agent

  1. Execute the following command to request an AWS credential.

    $ vault read aws/creds/readonly
    
    Key                Value
    ---                -----
    lease_id           aws/creds/readonly/oWBfLT2nIhXq9ZgWaC2e1fHy
    lease_duration     1h
    lease_renewable    true
    access_key         AKIAJMRXXXXXXXXMAGIA
    secret_key         swNXag6aVaultHappyYayAaivQ1iwMBaKCvuDoI
    security_token     <nil>
    
  2. See the logs in the terminal where Vault Agent is running.

    ...
    [INFO]  cache: received request: path=/v1/aws/creds/readonly method=GET
    [DEBUG] cache: using auto auth token: path=/v1/aws/creds/readonly method=GET
    [DEBUG] cache.leasecache: forwarding request: path=/v1/aws/creds/readonly method=GET
    [INFO]  cache.apiproxy: forwarding request: path=/v1/aws/creds/readonly method=GET
    [DEBUG] cache.leasecache: processing lease response: path=/v1/aws/creds/readonly method=GET
    [DEBUG] cache.leasecache: storing response into the cache: path=/v1/aws/creds/readonly method=GET
    [DEBUG] cache.leasecache: initiating renewal: path=/v1/aws/creds/readonly method=GET
    [DEBUG] cache.leasecache: secret renewed: path=/v1/aws/creds/readonly
    

    The log indicates that auto-auth token was used to connect Vault. The received request was forwarded to the Vault server, and the returned response was cached (find an entry storing response into the cache in the log).

    Benefit of Vault Agent: Vault Agent will manage the lifecycle of cached tokens and leases automatically so that the clients do not need to implement a logic to renew the tokens and leases.

  3. Re-run the vault command to study the agent behavior.

    $ vault read aws/creds/readonly
    
    Key                Value
    ---                -----
    lease_id           aws/creds/readonly/oWBfLT2nIhXq9ZgWaC2e1fHy
    lease_duration     1h
    lease_renewable    true
    access_key         AKIAJMRXXXXXXXXMAGIA
    secret_key         swNXag6aVaultHappyYayAaivQ1iwMBaKCvuDoI
    security_token     <nil>
    

    The agent log indicates:

    [INFO]  cache: received request: path=/v1/aws/creds/readonly method=GET
    [DEBUG] cache: using auto auth token: path=/v1/aws/creds/readonly method=GET
    [DEBUG] cache.leasecache: returning cached response: path=/v1/aws/creds/readonly
    

    This returns the same AWS credentials. Notice that the request_id and lease_id match to the ones you received the first time.

  4. Log in as a user, student via agent whose password is "pAssw0rd".

    # Login with username 'student' and password is "pAssw0rd"
    $ vault login -method=userpass username="student" password="pAssw0rd"
    
    Success! You are now authenticated. The token information displayed below
    is already stored in the token helper. You do NOT need to run "vault login"
    again. Future Vault requests will automatically use this token.
    
    Key                    Value
    ---                    -----
    token                  s.AKsRLji9mPq4xNZP1UIJggk6
    token_accessor         lU69kHGAPtYMoQndeCSoYuhs
    token_duration         48h
    token_renewable        true
    token_policies         ["default" "myapp"]
    ...
    
    # Store the acquired token in VAULT_TOKEN environment variable
    $ export VAULT_TOKEN="s.AKsRLji9mPq4xNZP1UIJggk6"
    

    Examine the agent log in the other terminal.

    ...
    [INFO]  cache: received request: path=/v1/auth/userpass/login/student method=POST
    [DEBUG] cache: using auto auth token: path=/v1/auth/userpass/login/student method=POST
    [DEBUG] cache.leasecache: forwarding request: path=/v1/auth/userpass/login/student method=POST
    [INFO]  cache.apiproxy: forwarding request: path=/v1/auth/userpass/login/student method=POST
    [DEBUG] cache.leasecache: processing auth response: path=/v1/auth/userpass/login/student method=POST
    [DEBUG] cache.leasecache: storing response into the cache: path=/v1/auth/userpass/login/student method=POST
    [DEBUG] cache.leasecache: initiating renewal: path=/v1/auth/userpass/login/student method=POST
    [DEBUG] cache.leasecache: secret renewed: path=/v1/auth/userpass/login/student
    ...
    

    The log should indicate that the acquired token for user student is successfully cached ([DEBUG] cache.leasecache: storing response into the cache). This is because the login request was sent through the agent.

  5. Create a new token with TTL set to 12 minutes.

    $ vault token create -ttl=12m
    
    Key                  Value
    ---                  -----
    token                s.e1h5wPbZ7i6FLCjPDqpsfDBG
    token_accessor       y8K7xJr15fFjSowWW1t3VERT
    token_duration       12m
    ...
    

    Examine the agent log in the other terminal.

    ...
    [INFO]  cache: received request: path=/v1/auth/token/create method=POST
    [DEBUG] cache.leasecache: forwarding request: path=/v1/auth/token/create method=POST
    [INFO]  cache.apiproxy: forwarding request: path=/v1/auth/token/create method=POST
    [DEBUG] cache.leasecache: processing auth response: path=/v1/auth/token/create method=POST
    [DEBUG] cache.leasecache: setting parent context: path=/v1/auth/token/create method=POST
    [DEBUG] cache.leasecache: storing response into the cache: path=/v1/auth/token/create method=POST
    ...
    

    This time, instead of using the auto-auth token, the command is using the student token generated upon successful login. The resulting token was successfully cached.

    NOTE: The agent will automatically renew the token to keep it valid:

    [DEBUG] cache.leasecache: renewal received; updating cache: path=/v1/auth/token/create
    
  6. Re-send the token create request.

    $ vault token create -ttl=12m
    
    Key                  Value
    ---                  -----
    token                s.e1h5wPbZ7i6FLCjPDqpsfDBG
    token_accessor       y8K7xJr15fFjSowWW1t3VERT
    ...
    

    The same token should be returned, and in the agent log, find the following:

    [INFO]  cache: received request: path=/v1/auth/token/create method=POST
    [DEBUG] cache.leasecache: returning cached response: path=/v1/auth/token/create
    

»Step 5: Cache Evictions

The eviction of cache will occur when the agent fails to renew leases or tokens. This can happen when the cached lease/token hits it's maximum TTL or if the renewal results in an error. Agent also does some best-effort cache evictions by observing specific request types and response codes.

While agent observes requests and evicts cached entries automatically, you can trigger a cache eviction by invoking the /agent/v1/cache-clear endpoint.

  1. If you need to manually evict a stale lease, invoke the /agent/v1/cache-clear endpoint with lease ID for which you wish to evict from the cache.

    Example:

    $ curl --request POST \
           --data '{ "type": "lease", "value": "aws/creds/readonly/8DLXo8nsgcIpx7iQPpCvWtlc" }' \
           $VAULT_AGENT_ADDR/agent/v1/cache-clear
    

    The agent log should show the following:

    [DEBUG] cache.leasecache: received cache-clear request: type=lease namespace= value=aws/creds/readonly/8DLXo8nsgcIpx7iQPpCvWtlc
    [DEBUG] cache.leasecache: canceling context of index attached to accessor
    [DEBUG] cache.leasecache: successfully cleared matching cache entries
    [DEBUG] cache.leasecache: context cancelled; stopping renewer: path=/v1/aws/creds/readonly
    [DEBUG] cache.leasecache: evicting index from cache: id=dd0f9e775... path=/v1/aws/creds/readonly method=GET
    
  2. Let's see what happens when you revoke a token.

    $ vault token revoke s.AkdB7Rgr0t8mIexWEGHPsNfn
    

    Or, revoke a token via API.

    $ curl --request POST \
           --data '{"token": "s.AkdB7Rgr0t8mIexWEGHPsNfn"}' \
           $VAULT_AGENT_ADDR/v1/auth/token/revoke
    

    Examine the agent log:

    ...
    [INFO]  cache: received request: path=/v1/auth/token/revoke method=POST
    [DEBUG] cache: using auto auth token: path=/v1/auth/token/revoke method=POST
    [DEBUG] cache.leasecache: forwarding request: path=/v1/auth/token/revoke method=POST
    [INFO]  cache.apiproxy: forwarding request: path=/v1/auth/token/revoke method=POST
    [DEBUG] cache.leasecache: cancelling context of index attached to token
    [DEBUG] cache.leasecache: successfully cleared matching cache entries
    [DEBUG] cache.leasecache: triggered caching eviction from revocation request
    [DEBUG] cache.leasecache: context cancelled; stopping renewer: path=/v1/auth/token/create
    [DEBUG] cache.leasecache: evicting index from cache: id=b5715bdca771174... path=/v1/auth/token/create method=POST
    

    In the agent log, notice the message, "triggered caching eviction from revocation request". When a token revocation request is made via the agent, the agent evicts the cached entries associated with the revoked token.

    Similarly, if you revoked a staled AWS lease, the agent will automatically evict the cache.

  3. If a situation requires you to clear all cached tokens and leases (e.g. reset after a number of testing), set the type to all.

    $ curl --request POST --data '{ "type": "all" }' \
           $VAULT_AGENT_ADDR/agent/v1/cache-clear
    

»Step 6: Clean Up

  1. On the server SSH terminal, execute the following command to revoke all leases.

    $ vault lease revoke -prefix aws/creds/readonly
    
  2. Execute the following commands to destroy cloud resources.

    $ terraform destroy -force
    
  3. Delete state files.

    $ rm -rf .terraform terraform.tfstate*
    

»Server Side Request Forgery (SSRF) Protection

NOTE: Vault 1.3 introduced Server Side Request Forgery (SSRF) protection for Vault Agent. To leverage this feature, download Vault 1.3 or later.

Add require_request_header = true in the Agent's listener configuration stanza to enable the SSRF protection.

Example:

exit_after_auth = false
pid_file = "./pidfile"

auto_auth {
   method "aws" {
       mount_path = "auth/aws"
       config = {
           type = "iam"
           role = "app-role"
       }
   }

   sink "file" {
       config = {
           path = "/home/ubuntu/vault-token-via-agent"
       }
   }
}

cache {
   use_auto_auth_token = true
}

listener "tcp" {
   address = "127.0.0.1:8200"
   require_request_header = true
}

vault {
   address = "http://<vault-server-host>:8200"
}

Notice the listener block.

When the require_request_header is set to true, the Agent listener will reject all requests that do NOT have the proper X-Vault-Request: true header.

For example, the command to request AWS credentials you saw in Step 4 must look like:

$ curl -header "X-Vault-Request: true" \
       $VAULT_AGENT_ADDR/v1/aws/creds/readonly

With absence of the X-Vault-Request header, the Agent will throw missing 'X-Vault-Request' header error, and the request will not be propagated to the Vault server.

»Help and Reference

  • Secure Introduction of Vault Clients
  • Vault Agent Auto-Auth
  • Vault Agent Caching
  • AWS Auth Method
  • Vault 1.1: Secret Caching with Vault Agent and Other New Features


Back to Collection
HashiCorp
  • System Status
  • Terms of Use
  • Security
  • Privacy
stdin: is not a tty