HashiConf
Join us this September for 3 days of talks, training, product news & more. Book Your Ticket Now

Identity and Access Management

Vault Agent with AWS

Nearly all requests to Vault must be accompanied by an authentication token. This includes all API requests, as well as via the Vault CLI and other libraries.

Vault provides a number of different authentication methods to assist in delivery of this initial token. If you can securely get the first secret from an originator to a consumer, all subsequent secrets transmitted between this originator and consumer can be authenticated with the trust established by the successful distribution and user of that first secret. Getting the first secret to the consumer, is the secure introduction challenge.

Secure Introduction

To that end, Vault provides integration with native authentication capabilities in various environments, for example: IAM in AWS and Google Cloud, Managed Service Identities in Azure, and Service Accounts in Kubernetes.

Challenge

Although a number of auth methods are available, the client is still responsible for managing the lifecycle of its Vault tokens. Therefore, the challenge becomes how to enable authentication to Vault and manage the lifecycle of tokens in a standard way without having to write custom logic.

Solution

Vault Agent provides a number of different helper features, specifically addressing the following challenges:

  • Automatic authentication
  • Secure delivery/storage of tokens
  • Lifecycle management of these tokens (renewal & re-authentication)

NOTE: The Vault Agent Auto-Auth functionality addresses the challenges related to obtaining and managing authentication tokens (secret zero).

Prerequisites

To complete this section of the guide, you will need an AWS account and associated credentials that allow for the creation of resources.

Download demo assets

Clone or download the demo assets from the hashicorp/vault-guides GitHub repository to perform the steps described in this guide.

Steps

You are going to perform the following steps:

  1. Provision the Cloud Resources
  2. Configure AWS IAM Auth Method
  3. Verify the AWS auth method configuration
  4. Leverage Vault Agent Auto-Auth
  5. Response Wrap the token
  6. Clean up

Step 1: Provision the Cloud Resources

Be sure to set your working directory to where the /identity/vault-agent-demo/terraform-aws folder is located.

The working directory should contain the provided Terraform files: Assets

NOTE: The example Terraform in this repository is created for the demo purpose, and not suitable for production use. For production deployment, refer the following examples:

  1. Set your AWS credentials as environment variables:

    $ export AWS_ACCESS_KEY_ID = "<YOUR_AWS_ACCESS_KEY_ID>"
    
    $ export AWS_SECRET_ACCESS_KEY = "<YOUR_AWS_SECRET_ACCESS_KEY>"
    
  2. Create a file named terraform.tfvars and specify the key_name. In addition, specify the variable values (variables.tf) which you wish to overwrite its default. (Use the provided terraform.tfvars.example as a base.)

    Example terrafrom.tfvars:

    # SSH key name to access EC2 instances (should already exist)
    key_name = "vault-test"
    
    # All resources will be tagged with this
    environment_name = "va-demo"
    
    # If you want to use a different AWS region
    aws_region = "us-west-1"
    availability_zones = "us-west-1a"
    
  3. Perform a terraform init to pull down the necessary provider resources. Then terraform plan to verify your changes and the resources that will be created. If all looks good, then perform a terraform apply to provision the resources.

    $ terraform init
    Initializing provider plugins...
    ...
    Terraform has been successfully initialized!
    
$ terraform plan
...
Plan: 20 to add, 0 to change, 2 to destroy.


$ terraform apply
...
Apply complete! Resources: 20 added, 0 changed, 2 destroyed.

Outputs:

endpoints =
Vault Server IP (public):  54.219.129.15
Vault Server IP (private): 10.0.101.50

For example:
   ssh -i vault-test.pem ubuntu@54.219.129.15

Vault Client IP (public):  54.183.212.51
Vault Client IP (private): 10.0.101.209

For example:
   ssh -i vault-test.pem ubuntu@54.183.212.51
```

The Terraform output will display the public IP address to SSH into your Vault
**server** and **client** instances.

Step 2: Configure AWS IAM Auth Method

In this step, you will configure Vault to allow AWS IAM authentication from specific IAM roles.

  1. SSH into the Vault Server instance.

    Output

    ssh -i <path_to_key> ubuntu@<public_ip_of_server>
    ...
    Are you sure you want to continue connecting (yes/no)? yes
    

    When you are prompted, enter "yes" to continue.

  2. To verify that Vault has been installed, run vault status command and notice that Initialized is false.

    $ vault status
    Key                      Value
    ---                      -----
    Recovery Seal Type       awskms
    Initialized              false
    Sealed                   true
    Total Recovery Shares    0
    Threshold                0
    Unseal Progress          0/0
    Unseal Nonce             n/a
    Version                  n/a
    HA Enabled               true
    
  3. Run the vault operator init command to initialize the Vault server:

    $ vault operator init
    
    ...
    
    Initial Root Token: s.20JnHBY66EKTj9zyR6SjTMNq
    
    Success! Vault is initialized
    ...
    
  4. Copy the Initial Root Token value.

  5. Log into Vault using the generated initial root token:

    $ vault login s.20JnHBY66EKTj9zyR6SjTMNq
    
    Success! You are now authenticated. The token information displayed below
    is already stored in the token helper. You do NOT need to run "vault login"
    again. Future Vault requests will automatically use this token.
    
    Key                  Value
    ---                  -----
    token                s.20JnHBY66EKTj9zyR6SjTMNq
    token_accessor       5mRVTsVHR8zx2ajUDWhYyxmQ
    token_duration       ∞
    token_renewable      false
    token_policies       ["root"]
    identity_policies    []
    policies             ["root"]
    
  6. Write some secrets at secret/myapp/config, and create myapp policy which grants read-only permission on the secret/myapp/* path:

    # Create a policy file
    $ tee myapp.hcl <<EOF
    path "secret/myapp/*" {
        capabilities = ["read", "list"]
    }
    EOF
    
    # Create a policy named, 'myapp'
    $ vault policy write myapp myapp.hcl
    
    # Enable K/V secrets engine at 'secret/'
    $ vault secrets enable -path="secret" kv
    
    # Write some secrets in 'secret/app/config' path
    $ vault kv put secret/myapp/config \
        ttl='30s' \
        username='appuser' \
        password='suP3rsec(et!'
    
  7. Enable the aws auth method:

    $ vault auth enable aws
    
    Success! Enabled aws auth method at: aws/
    
  8. Configure the AWS credentials that Vault will use to verify login requests from AWS clients:

    $ vault write -force auth/aws/config/client
    
  9. Log in to the Amazon EC2 Dashboard, and select Instances. In Step 1, Terraform provisioned a Vault client instance ("${var.environment_name}-vault-client"). Select the client instance, and click on its IAM role name. Secure Introduction

  10. Copy the Role ARN. Secure Introduction

  11. In the Vault server SSH session, execute the following command to configure a client role where <ROLE_ARN> is the IAM role ARN you copied.

    $ vault write auth/aws/role/dev-role-iam auth_type=iam \
            bound_iam_principal_arn=<ROLE_ARN> \
            policies=myapp \
            ttl=24h
    
    Success! Data written to: auth/aws/role/dev-role-iam
    

    This creates dev-role-iam role which has myapp policy attached.

Step 3: Verify the AWS auth method configuration

Now that you have configured the appropriate AWS IAM auth method on our Vault server, let's SSH into the client instance and verify that you can utilize the instance profile to login to Vault.

  1. Now, SSH into the Vault Client instance.

    Output

    ssh -i <path_to_key> ubuntu@<public_ip_of_client>
    ...
    Are you sure you want to continue connecting (yes/no)? yes
    

    When you are prompted, enter "yes" to continue.

  2. Vault binary is already installed and configured to talk to your Vault server on the client instance.

    $ vault status
    
    Key                      Value
    ---                      -----
    Recovery Seal Type       shamir
    Initialized              true
    Sealed                   false
    Total Recovery Shares    5
    Threshold                3
    Version                  1.1.0
    Cluster Name             vault-cluster-f846a973
    Cluster ID               eae2cbc6-d358-6f80-14bf-82d98d3945e7
    HA Enabled               true
    HA Cluster               https://10.0.101.53:8201
    HA Mode                  active
    
  3. Log into Vault using the AWS auth method:

    $ vault login -method=aws role=dev-role-iam
    
    Success! You are now authenticated. The token information displayed below
    is already stored in the token helper. You do NOT need to run "vault login"
    again. Future Vault requests will automatically use this token.
    
    Key                                Value
    ---                                -----
    token                              s.4uwdsqcVKNd9ZfAEYraX2PX2
    token_accessor                     8HWlpAvycJ3uxB34rxGFLM97
    token_duration                     24h
    token_renewable                    true
    token_policies                     ["default" "myapp"]
    identity_policies                  []
    policies                           ["default" "myapp"]
    token_meta_inferred_aws_region     n/a
    token_meta_inferred_entity_id      n/a
    token_meta_inferred_entity_type    n/a
    token_meta_account_id              CLIENT_USER_ID_VALUE
    token_meta_auth_type               iam
    token_meta_canonical_arn           arn:aws:iam::AWS_ACCOUNT_NUMBER:role/va-demo-vault-client-role
    token_meta_client_arn              arn:aws:sts::AWS_ACCOUNT_NUMBER:assumed-role/va-demo-vault-client-role/i-073f...
    token_meta_client_user_id          AROAICHLS7LVRFEUU2KIA
    
  4. Check to make sure that the token has the appropriate permissions to read the secrets:

    $ vault kv get secret/myapp/config
    
    ====== Data ======
    Key         Value
    ---         -----
    password    suP3rsec(et!
    ttl         30s
    username    appuser
    

Step 4: Leverage Vault Agent Auto-Auth

Now, you are going to see how Vault Agent Auto-Auth method works, and write out a token to an arbitrary location on disk. Vault Agent is a client daemon and its Auto-Auth feature allows for easy authentication to Vault.

Vault Agent

NOTE: To run Vault Agent, you need Vault 0.11 or later.

  1. In the client instance, explorer the Vault Agent configuration file (/home/ubuntu/vault-agent.hcl).

    $ cat vault-agent.hcl
    
    exit_after_auth = true
    pid_file = "./pidfile"
    
    auto_auth {
       method "aws" {
           mount_path = "auth/aws"
           config = {
               type = "iam"
               role = "dev-role-iam"
           }
       }
    
       sink "file" {
           config = {
               path = "/home/ubuntu/vault-token-via-agent"
           }
       }
    }
    
    vault {
       address = "http://10.0.101.53:8200"
    }
    

    The top level auto_auth block has two configuration entries: method and sinks. In this example, the Auto-Auth is configured to use the aws auth method enabled at the auth/aws path on the Vault server. The Vault Agent will use the dev-role-iam role to authenticate.

    The sink block specifies the location on disk where to write tokens. Vault Agent Auto-Auth sink can be configured multiple times if you want Vault Agent to place the token into multiple locations. In this example, the sink is set to /home/ubuntu/vault-token-via-agent.

  2. Execute the following command to get help:

    $ vault agent -h
    
  3. Now, you are ready to run the Vault Agent. Execute the following command:

    $ vault agent -config=/home/ubuntu/vault-agent.hcl -log-level=debug
    
    ==> Vault agent configuration:
    
                    Cgo: disabled
              Log Level: debug
                Version: Vault v1.0.0
            Version Sha: c19cef14891751a23eaa9b41fd456d1f99e7e856
    
    ==> Vault server started! Log data will stream in below:
    
    2018-11-20T00:26:23.439Z [INFO]  sink.file: creating file sink
    2018-11-20T00:26:23.439Z [INFO]  sink.file: file sink configured: path=/home/ubuntu/vault-token-via-agent
    2018-11-20T00:26:23.442Z [INFO]  sink.server: starting sink server
    2018-11-20T00:26:23.442Z [INFO]  auth.handler: starting auth handler
    2018-11-20T00:26:23.442Z [INFO]  auth.handler: authenticating
    2018-11-20T00:26:23.813Z [INFO]  auth.handler: authentication successful, sending token to sinks
    2018-11-20T00:26:23.813Z [INFO]  auth.handler: starting renewal process
    2018-11-20T00:26:23.814Z [INFO]  sink.file: token written: path=/home/ubuntu/vault-token-via-agent
    2018-11-20T00:26:23.814Z [INFO]  sink.server: sink server stopped
    2018-11-20T00:26:23.814Z [INFO]  sinks finished, exiting
    
  4. Returned token is written in the /home/ubuntu/vault-token-via-agent file.

    $ more vault-token-via-agent
    s.6Wqj0YNiRngCCf6Cic3dA5Ul
    
  5. Let's try an API call using the token that Vault Agent pulled for us to test:

    $ curl --header "X-Vault-Token: $(cat /home/ubuntu/vault-token-via-agent)" \
          $VAULT_ADDR/v1/secret/myapp/config | jq
    {
      ...
      "data": {
        "password": "suP3rsec(et!",
        "ttl": "30s",
        "username": "appuser"
      },
      ...
    }
    

Step 5: Response Wrap the token

It may not be ideal to write the token as a plaintext depending on the firewall around the client instance. Vault Agent supports response-wrapping of the token to provide an additional layer of protection for the token. Tokens can be wrapped by either the auth method or by the sink configuration, with each approach solving for different challenges as described in Response-Wrapping Tokens.

  1. In the client instance, explorer the /home/ubuntu/vault-agent-wrapped.hcl file which supports response-wrapping of the token.

    $ cat /home/ubuntu/vault-agent-wrapped.hcl
    
    exit_after_auth = true
    pid_file = "./pidfile"
    
    auto_auth {
        method "aws" {
            mount_path = "auth/aws"
            config = {
                type = "iam"
                role = "dev-role-iam"
            }
        }
    
        sink "file" {
            wrap_ttl = "5m"
            config = {
                path = "/home/ubuntu/vault-token-via-agent"
            }
        }
    }
    
    vault {
       address = "http://10.0.101.53:8200"
    }
    
  2. From the client, run the Vault Agent again and inspect the output:

    $ vault agent -config=/home/ubuntu/vault-agent-wrapped.hcl -log-level=debug
    ==> Vault agent configuration:
    
                     Cgo: disabled
               Log Level: debug
                 Version: Vault v1.0.0
             Version Sha: c19cef14891751a23eaa9b41fd456d1f99e7e856
    
    ==> Vault server started! Log data will stream in below:
    
    2018-11-20T00:58:14.993Z [INFO]  sink.file: creating file sink
    2018-11-20T00:58:14.993Z [INFO]  sink.file: file sink configured: path=/home/ubuntu/vault-token-via-agent
    2018-11-20T00:58:14.995Z [INFO]  sink.server: starting sink server
    2018-11-20T00:58:14.995Z [INFO]  auth.handler: starting auth handler
    2018-11-20T00:58:14.995Z [INFO]  auth.handler: authenticating
    2018-11-20T00:58:15.262Z [INFO]  auth.handler: authentication successful, sending token to sinks
    2018-11-20T00:58:15.263Z [INFO]  auth.handler: starting renewal process
    2018-11-20T00:58:15.269Z [INFO]  auth.handler: renewed auth token
    2018-11-20T00:58:15.281Z [INFO]  sink.file: token written: path=/home/ubuntu/vault-token-via-agent
    2018-11-20T00:58:15.281Z [INFO]  sink.server: sink server stopped
    2018-11-20T00:58:15.281Z [INFO]  sinks finished, exiting
    
    $ cat /home/ubuntu/vault-token-via-agent | jq
    {
      "token": "s.7h0DPYB4GzVlqxP3JbhDzDcV",
      "accessor": "1k1j6ieCnXyX9UTDZQ9M7ST0",
      "ttl": 300,
      "creation_time": "2018-11-20T00:58:15.270524147Z",
      "creation_path": "sys/wrapping/wrap",
      "wrapped_accessor": ""
    }
    

    Instead of a token value, you now have a JSON object containing a wrapping token as well as some additional metadata. In order to get to the true token, you need to first perform an unwrap operation.

  3. Unwrap the response-wrapped token and save it to a VAULT_TOKEN env var that other applications can use:

    # Execute 'vault unwrap' command and set the resulting token as VAULT_TOKEN
    $ export VAULT_TOKEN=$(vault unwrap -field=token $(jq -r '.token' /home/ubuntu/vault-token-via-agent))
    
# See the unwrapped token value
$ echo $VAULT_TOKEN
s.3VaVYxv0rNgb5JLZBQqS8BXX


# Test to make sure that the token has the read permission on 'secret/myapp/config'
$ curl --header "X-Vault-Token: $VAULT_TOKEN" $VAULT_ADDR/v1/secret/myapp/config | jq
```

Notice that the value saved to the `VAULT_TOKEN` is not the same as the
`token` value in the `/home/ubuntu/vault-token-via-agent` file. The value in
`VAULT_TOKEN` is the unwrapped token retrieved by Vault Agent.


If you try to unwrap that same value again or wait longer than 5 minutes
(`wrap_ttl`), it will throw an error:

```plaintext
$ export VAULT_TOKEN=$(vault unwrap -field=token $(jq -r '.token' /home/ubuntu/vault-token-via-agent))

Error unwrapping: Error making API request.

URL: PUT http://active.vault-va-demo.service.dc1.consul:8200/v1/sys/wrapping/unwrap
Code: 400. Errors:

* wrapping token is not valid or does not exist
```

A response-wrapped token can only be unwrapped once. Additional attempts to
unwrap an already-unwrapped token will result in triggering an error.

Step 6: Clean Up

Once completed, execute the following commands to clean up:

$ terraform destroy -force

$ rm -rf .terraform terraform.tfstate*

Additional Discussion

In the above examples, you manually ran Vault Agent in order to demonstrate how it works. How you actually integrate Vault Agent into your application deployment workflow will vary with a number of factors. Some questions to ask to help determine appropriate usage:

  • What is the lifecycle of my application? Is it more ephemeral or long-lived?
  • What are the lifecycles of my authentication tokens? Are they long-lived and simply need to be renewed over and over to demonstrate liveliness of a service or do you want to enforce periodic re-authentications?
  • Do I have a number of applications running on my host that each needs their own token? Can I use a native authentication capability (e.g. AWS IAM, K8s, Azure MSI, Google Cloud IAM, etc.)?

The answers to these questions will help you determine if Vault Agent should run as a daemon or as a prerequisite of a service configuration. Take for example the following Systemd service definition for running Nomad:

[Unit]
Description=Nomad Agent
Requires=consul-online.target
After=consul-online.target

[Service]
KillMode=process
KillSignal=SIGINT
Environment=VAULT_ADDR=http://active.vault.service.consul:8200
Environment=VAULT_SKIP_VERIFY=true
ExecStartPre=/usr/local/bin/vault agent -config /etc/vault-agent.d/vault-agent.hcl
ExecStart=/usr/bin/nomad-vault.sh
ExecReload=/bin/kill -HUP $MAINPID
Restart=on-failure
RestartSec=2
StartLimitBurst=3
StartLimitIntervalSec=10
LimitNOFILE=65536

[Install]
WantedBy=multi-user.target

Notice the ExecStartPre directive that runs Vault Agent before the desired service starts. The service startup script expects a Vault token to be set as is demonstrated in the /usr/bin/nomad-vault.sh startup script:

#!/usr/bin/env bash

if [ -f /mnt/ramdisk/token ]; then
  exec env VAULT_TOKEN=$(vault unwrap -field=token $(jq -r '.token' /mnt/ramdisk/token)) \
    /usr/local/bin/nomad agent \
      -config=/etc/nomad.d \
      -vault-tls-skip-verify=true
else
  echo "Nomad service failed due to missing Vault token"
  exit 1
fi

Many applications that expect Vault tokens typically look for a VAULT_TOKEN env var. Here, you're using Vault Agent to obtain a token and write it out to a Ramdisk and as part of the Nomad startup script, you read the response-wrapped token from the Ramdisk and save it to our VAULT_TOKEN env var before actually starting Nomad.

Furthermore, since Vault Agent Auto-Auth only addresses the challenges of obtaining and managing authentication tokens (secret zero), you might want to use helper tools such as Consul Template and Envconsul to obtain secrets stored in Vault (e.g. DB credentials, PKI certificates, AWS access keys, etc.).

Help and Reference

To learn more about the response wrapping feature, refer the following: