Virtual Day
Building the ecosystem for the Cloud Operating Model with Azure, Cisco, F5 and GitLab Register

Access Management

Vault Agent Templates

The adoption of Vault is an incremental journey. First, you move your secrets into Vault so that they are securely encrypted and stored. The next step is to update your applications' behavior so that the secrets are read from Vault.

To establish a connection with Vault, clients must first authenticate with Vault and acquire a token. Vault Agent was introduced to reduce the burden from distributed applications to manage Vault client tokens.

Vault Agent

The following guides demonstrate the Auto-Auth method of Vault Agent:

In addition, the Vault Agent Caching guide walked through the Caching feature introduced in Vault 1.1.

»Challenge & Solution

After the successful authentication, Vault clients can start interacting with the Vault. Many Vault users adopted the Consul Template tool to minimize the level of changes introduced to their existing applications.

Vault and Consul Tempalte

The Direct Application Integration guide demonstrates the use of Consul Template for Vault users.

In this case, Consul Template is the client directly interacting with Vault; therefore, you had to configure Consul Template to read the token from the agent's sink location in order to interact with Vault. This requires you to operate two distinct tools to provide secrets to applications.

In Vault 1.3, Vault Agent introduced Vault Agent Templates allowing Vault secrets to be rendered to files using the Consul Template markup language. This significantly simplifies the workflow when you are integrating your applications with Vault.

»Prerequisites

To complete this section of the guide, you will need:

  • An AWS account and associated credentials that allow for the creation of resources
  • Vault version 1.3 or later

NOTE: An interactive tutorial is also available if you do not have a Vault environment to perform the steps described in this guide. Click the Show Tutorial button to launch the tutorial.

»Download demo assets

Clone or download the demo assets from the hashicorp/vault-guides GitHub repository to perform the steps described in this guide.

Clone the repository:

$ git clone https://github.com/hashicorp/vault-guides.git

Or download the repository:

Download

This repository contains supporting content for all of the Vault learn guides. The content specific to this guide can be found within a sub-directory.

»Step 1: Provision the Cloud Resources

Be sure to set your working directory to where the /identity/vault-agent-templates/terraform-aws directory is located.

$ cd vault-guides/identity/vault-agent-templates/terraform-aws

The following assets are located under this directory:

$ tree
.
├── aws.tf
├── iam.tf
├── kms.tf
├── network.tf
├── outputs.tf
├── security-groups.tf
├── templates
│   ├── userdata-vault-client.tpl
│   └── userdata-vault-server.tpl
├── terraform.tfvars.example
├── variables.tf
├── vault-client.tf
├── vault-server.tf
└── versions.tf

NOTE: The example Terraform in this repository is created for the demo purpose.

  1. Set your AWS credentials as environment variables.

    $ export AWS_ACCESS_KEY_ID = "<YOUR_AWS_ACCESS_KEY_ID>"
    $ export AWS_SECRET_ACCESS_KEY = "<YOUR_AWS_SECRET_ACCESS_KEY>"
    
  2. Create a file named terraform.tfvars and specify the key_name. In addition, specify the variable values (variables.tf) which you wish to overwrite its default. (Use the provided terraform.tfvars.example as a base.)

    Example terraform.tfvars:

    # SSH key name to access EC2 instances (should already exist) on the AWS region
    key_name = "my-key-pair"
    

    If you don't have an EC2 Key Pairs, refer to the AWS documentation and create one.

  3. Perform a terraform init to pull down the necessary provider resources.

    $ terraform init
    
  4. Perform terraform plan to verify your changes and the resources that will be created.

    $ terraform plan
    
  5. If all looks good, perform a terraform apply to provision the resources.

    $ terraform apply -auto-approve
    ...
    Apply complete! Resources: 20 added, 0 changed, 0 destroyed.
    
    Outputs:
    
    endpoints =
    Vault Server IP (public):  192.0.2.224
    Vault Server IP (private): 10.0.101.62
    
    For example:
      ssh -i vault-test.pem ubuntu@192.0.2.224
    
    Vault Client IP (public):  198.51.100.24
    Vault Client IP (private): 10.0.101.10
    
    For example:
      ssh -i vault-test.pem ubuntu@198.51.100.24
    

    The Terraform output will display the public IP address to SSH into your Vault server and client instances.

»Step 2: Configure AWS IAM Auth Method

In this step, you will configure Vault to allow AWS IAM authentication from specific IAM roles.

  1. SSH into the Vault Server instance.

    Output

    ssh -i <path_to_key> ubuntu@<public_ip_of_server>
    ...
    Are you sure you want to continue connecting (yes/no)? yes
    

    When you are prompted, enter "yes" to continue.

  2. Run the vault operator init command to initialize the Vault server. For the convenience of this tutorial, save the generated recovery keys and an initial root token in a file named, key.txt.

    $ vault operator init > key.txt
    

    The Vault server is configured to auto-unseal with AWS Key Management Service (KMS); therefore, once the server is initialized, it gets automatically unsealed. To learn how to auto-unseal Vault using AWS KMS, refer to the Auto-unseal using AWS KMS guide.

  3. Log into Vault using the generated initial root token save in the key.txt

    $ vault login $(grep 'Initial Root Token:' key.txt | awk '{print $NF}')
    
  4. Examine the /home/ubuntu/aws_auth.sh script.

    $ cat aws_auth.sh
    
  5. Execute the /home/ubuntu/aws_auth.sh script.

    $ ./aws_auth.sh
    
    Success! Enabled the kv-v2 secrets engine at: secret/
    Key              Value
    ---              -----
    created_time     2019-12-04T00:16:07.107076447Z
    deletion_time    n/a
    destroyed        false
    version          1
    Success! Uploaded policy: app-pol
    Success! Enabled aws auth method at: aws/
    Success! Data written to: auth/aws/config/client
    Success! Data written to: auth/aws/role/app-role
    

    The script enabled key/value v2 secrets engine at secret and wrote some data at secret/customers/acme. It also created an app-pol policy, enabled and configured aws auth method, and created a role named app-role.

  6. View the app-pol policy.

    $ vault policy read app-pol
    
    path "secret/data/*" {
      capabilities = ["read"]
    }
    path "auth/token/*" {
      capabilities = ["create", "update"]
    }
    
  7. View the secrets written at secret/customers/acme

    $ vault kv get secret/customers/acme
    ====== Metadata ======
    Key              Value
    ---              -----
    created_time     2019-12-04T00:16:07.107076447Z
    deletion_time    n/a
    destroyed        false
    version          1
    
    ======== Data ========
    Key              Value
    ---              -----
    contact_email    james@acme.com
    customer_id      ABXX2398YZPIE7391
    organization     ACME Inc.
    region           US-West
    status           active
    type             premium
    zip_code         94105
    

»Step 3: Start Vault Agent

In the client node, Vault Agent authenticates with Vault via aws auth method you configured in Step 2. Using the acquired token, the agent pulls secrets from Vault defined by the template file.

Scenario Diagram

  1. Now, SSH into the Vault Client instance.

    Output

    ssh -i <path_to_key> ubuntu@<public_ip_of_client>
    ...
    Are you sure you want to continue connecting (yes/no)? yes
    

    When you are prompted, enter "yes" to continue.

  2. Explore the Vault Agent configuration file, /home/ubuntu/vault-agent.hcl.

    $ cat vault-agent.hcl
    

    The contents of the file looks as below.

    pid_file = "./pidfile"
    
    auto_auth {
      method "aws" {
        mount_path = "auth/aws"
        config = {
          type = "iam"
          role = "app-role"
        }
      }
    
      sink "file" {
        config = {
          path = "/home/ubuntu/vault-token-via-agent"
        }
      }
    }
    
    vault {
      address = "http://10.0.101.62:8200"
    }
    
    template {
      source      = "/home/ubuntu/customer.tmpl"
      destination = "/home/ubuntu/customer.txt"
    }
    

    The Vault Agent uses the aws auth method to authenticate with the Vault server running on the server instance as app-role. Also notice that there is template block which sets /home/ubuntu/customer.tmpl as the source template file and the output will be written to the destination, /home/ubuntu/customer.txt.

  3. Execute the following command to start the Vault Agent with log level set to debug.

    $ vault agent -config=/home/ubuntu/vault-agent.hcl -log-level=debug
    

    You should find the following:

    [INFO]  sink.file: creating file sink
    [INFO]  sink.file: file sink configured: path=/home/ubuntu/vault-token-via-agent mode=-rw-r-----
    [INFO]  auth.handler: starting auth handler
    [INFO]  auth.handler: authenticating
    [INFO]  template.server: starting template server
    ...
    [INFO]  auth.handler: authentication successful, sending token to sinks
    ...
    [DEBUG] (runner) running initial templates
    [DEBUG] (runner) initiating run
    [DEBUG] (runner) checking template 56ba7f3e29857b85850ab5e0eb1151a3
    ...
    [DEBUG] (runner) receiving dependency vault.read(secret/data/customers/acme)
    [DEBUG] (runner) initiating run
    [DEBUG] (runner) checking template 56ba7f3e29857b85850ab5e0eb1151a3
    [DEBUG] (runner) rendering "/home/ubuntu/customer.tmpl" => "/home/ubuntu/customer.txt"
    
  4. Open a second SSH terminal into the client machine.

  5. Verify the client token stored in /home/ubuntu/vault-token-via-agent.

    $ more vault-token-via-agent
    s.4G9iuH8MtZqrvYtOmWeaGqnJ
    
  6. Check the details about the token (e.g. attached policies, TTL, etc.).

    $ VAULT_TOKEN="$(cat vault-token-via-agent)" vault token lookup
    Key                  Value
    ---                  -----
    ...
    orphan               true
    path                 auth/aws/login
    policies             [app-pol default]
    renewable            true
    ttl                  22h22m17s
    type                 service
    

    The app-pol policy is attached to the token. Remember that the app-pol policy grants read operation against the secret/data/* path.

  7. Let's review the customer.tmpl file which is written using the Consul Templates markup language.

    $ cat customer.tmpl
    
    {{ with secret "secret/data/customers/acme" }}
    Organization: {{ .Data.data.organization }}
    ID: {{ .Data.data.customer_id }}
    Contact: {{ .Data.data.contact_email }}
    {{ end }}
    

    Notice that the secret path is set to secret/data/customers/acme.

  8. Examine the resulting customer.txt file.

    $ cat customer.txt
    
    Organization: ACME Inc.
    ID: ABXX2398YZPIE7391
    Contact: james@acme.com
    

    Based on the customer.tmpl file, Vault Agent Templates read the secrets at secret/data/customers/acme and then generated the customer.txt file.

»Challenge

What happens if the data at secret/data/customers/acme was updated?

  1. On the server instance terminal, update the contact_email to jenn@acme.com.

    $ vault kv patch secret/customers/acme contact_email="jenn@acme.com"
    
  2. Verify that the data was successfully updated.

    $ vault kv get secret/customers/acme
    
  3. Return to the client instance SSH session where the Vault Agent is running. Wait for a few minutes (about 4 minutes). The agent pulls the secrets again.

    ...
    [DEBUG] (runner) checking template 494ea5cbe4765bfe2e5eca2363eff06b
    [DEBUG] (runner) rendering "./customer.tmpl" => "./customer.txt"
    [INFO] (runner) rendered "./customer.tmpl" => "./customer.txt"
    [DEBUG] (runner) diffing and updating dependencies
    [DEBUG] (runner) vault.read(secret/data/customers/acme) is still needed
    [DEBUG] (runner) watching 1 dependencies
    [DEBUG] (runner) all templates rendered
    ...
    

    When reading secrets from Key/Value v2 secrets engine, omitting the ?version parameter reads the latest version of the secrets at the path. If you want to always pull a specific version of the secret, specify the desired version number. For example, {{ with secret "secret/data/customers/acme?version=3" }} will always read the version 3 of the secret/data/customers/acme values.

»Step 4: Clean Up

Execute the following commands to destroy cloud resources.

$ terraform destroy -force

At this point, you can delete the Terraform state files from the directory.

$ rm -rf .terraform terraform.tfstate*

»Summary and Reference

Vault Agent is a client daemon that solves the secret-zero problem by authenticating with Vault and manage the client tokens on behalf of the client applications. The Consul Template tool is widely adopted by the Vault users since it allowed applications to be "Vault-unaware".

Vault Agent Templates combines the best of the two tools to make the end-to-end workflow even simpler.

Scenario Diagram

»Help and Reference