Skip to main content
June 20-22 Announcing HashiConf Europe full schedule: keynotes, sessions, labs & more Register Now
Type '/' to Search

Vault Agent with AWS

This tutorial also appears in: Auth Methods.

Nearly all requests to Vault must be accompanied by an authentication token. This includes all API requests, as well as via the Vault CLI and other libraries.

Vault provides a number of different authentication methods to assist in delivery of this initial token. If you can securely get the first secret from an originator to a consumer, all subsequent secrets transmitted between this originator and consumer can be authenticated with the trust established by the successful distribution and user of that first secret. Getting the first secret to the consumer, is the secure introduction challenge.

Secure Introduction

To that end, Vault provides integration with native authentication capabilities in various environments, for example: IAM in AWS and Google Cloud, Managed Service Identities in Azure, and Service Accounts in Kubernetes.

»Challenge

Although a number of auth methods are available, the client is still responsible for managing the lifecycle of its Vault tokens. Therefore, the challenge becomes how to enable authentication to Vault and manage the lifecycle of tokens in a standard way without having to write custom logic.

»Solution

Vault Agent provides a number of different helper features, specifically addressing the following challenges:

  • Automatic authentication
  • Secure delivery/storage of tokens
  • Lifecycle management of these tokens (renewal & re-authentication)

NOTE: The Vault Agent Auto-Auth functionality addresses the challenges related to obtaining and managing authentication tokens (secret zero).

The Secure Introduction of Vault Clients introduced three basic approaches: Platform Integration, Trusted Orchestrator, and Vault Agent. This tutorial demonstrates how Vault Agent works.

»Prerequisites

To complete this section of the tutorial, you will need:

»Provision the Cloud Resources

  1. Clone the demo assets from the hashicorp/vault-guides GitHub repository to perform the steps described in this tutorial.

    $ git clone https://github.com/hashicorp/vault-guides.git

    $ git clone https://github.com/hashicorp/vault-guides.git

    This repository contains supporting content for all of the Vault learn tutorials. The content specific to this tutorial can be found within a sub-directory.

  2. Be sure to set your working directory to where the /identity/vault-agent-demo/terraform-aws folder is located.

    $ cd vault-guides/identity/vault-agent-demo/terraform-aws

    $ cd vault-guides/identity/vault-agent-demo/terraform-aws

    The working directory should contain the provided Terraform files:

    $ tree
.
├── aws.tf
├── iam.tf
├── kms.tf
├── network.tf
├── outputs.tf
├── security-groups.tf
├── templates
│   ├── userdata-vault-client.tpl
│   └── userdata-vault-server.tpl
├── terraform.tfvars.example
├── variables.tf
├── vault-client.tf
├── vault-server.tf
└── versions.tf

1 directory, 13 files

    $ tree
.
├── aws.tf
├── iam.tf
├── kms.tf
├── network.tf
├── outputs.tf
├── security-groups.tf
├── templates
│   ├── userdata-vault-client.tpl
│   └── userdata-vault-server.tpl
├── terraform.tfvars.example
├── variables.tf
├── vault-client.tf
├── vault-server.tf
└── versions.tf

1 directory, 13 files

    NOTE: The example Terraform in this repository is created for the demo purpose, and not suitable for production use. For production deployment, refer the example Terraform in the /operations/provision-vault folder.

  3. Set an AWS_ACCESS_KEY_ID environment variable to hold your AWS access key ID.

    $ export AWS_ACCESS_KEY_ID = "<YOUR_AWS_ACCESS_KEY_ID>"

    $ export AWS_ACCESS_KEY_ID = "<YOUR_AWS_ACCESS_KEY_ID>"

  4. Set an AWS_SECRET_ACCESS_KEY environment variable to hold your AWS secret access key.

    $ export AWS_SECRET_ACCESS_KEY = "<YOUR_AWS_SECRET_ACCESS_KEY>"

    $ export AWS_SECRET_ACCESS_KEY = "<YOUR_AWS_SECRET_ACCESS_KEY>"

    Tip: The above example uses IAM user authentication. You can use any authentication method described in the AWS provider documentation.

  5. Create a file named terraform.tfvars and specify the key_name. (You can copy the terraform.tfvars.example file and rename it as terraform.tfvars.)

    Example:

    # SSH key name to access EC2 instances (should already exist)
key_name = "vault-test"
    terraform.tfvars
    # SSH key name to access EC2 instances (should already exist)
key_name = "vault-test"

    If you don't have an EC2 key pair, follow the AWS documentation to create one.

  6. Perform a terraform init to pull down the necessary provider resources.

    $ terraform init

Initializing provider plugins...
...snipped...
Terraform has been successfully initialized!

    $ terraform init

Initializing provider plugins...
...snipped...
Terraform has been successfully initialized!

  7. Run terraform apply and review the planned actions. Your terminal output should indicate the plan is running and what resources will be created.

    $ terraform apply

...snipped...
Plan: 20 to add, 0 to change, 0 to destroy.

Changes to Outputs:
  + endpoints = (known after apply)

Do you want to perform these actions?
  Terraform will perform the actions described above.
  Only 'yes' will be accepted to approve.

  Enter a value: yes

    $ terraform apply

...snipped...
Plan: 20 to add, 0 to change, 0 to destroy.

Changes to Outputs:
  + endpoints = (known after apply)

Do you want to perform these actions?
  Terraform will perform the actions described above.
  Only 'yes' will be accepted to approve.

  Enter a value: yes

    Enter yes to confirm and resume.

    When the apply command completes, the Terraform output will display the public IP address to SSH into your Vault server and client instances.

    Example output:

    Apply complete! Resources: 20 added, 0 changed, 0 destroyed.

Outputs:

endpoints =
Vault Server IP (public):  54.219.129.15
Vault Server IP (private): 10.0.101.50

For example:
  ssh -i vault-test.pem ubuntu@54.219.129.15

Vault Client IP (public):  54.183.212.51
Vault Client IP (private): 10.0.101.209

For example:
  ssh -i vault-test.pem ubuntu@54.183.212.51

    Apply complete! Resources: 20 added, 0 changed, 0 destroyed.

Outputs:

endpoints =
Vault Server IP (public):  54.219.129.15
Vault Server IP (private): 10.0.101.50

For example:
  ssh -i vault-test.pem ubuntu@54.219.129.15

Vault Client IP (public):  54.183.212.51
Vault Client IP (private): 10.0.101.209

For example:
  ssh -i vault-test.pem ubuntu@54.183.212.51

»Configure AWS IAM Auth Method

This step should be performed on the Vault Server instance.

In this step, you will configure Vault to allow AWS IAM authentication from specific IAM roles.

  1. SSH into the Vault Server instance.

    Output

    When you are prompted, enter "yes" to continue.

    $ ssh -i <path_to_key> ubuntu@<public_ip_of_server>
...
Are you sure you want to continue connecting (yes/no)? yes

    $ ssh -i <path_to_key> ubuntu@<public_ip_of_server>
...
Are you sure you want to continue connecting (yes/no)? yes

    NOTE: If you received Permissions 0664 for '<key_name>.pem' are too open error, be sure to set the file permission appropriately.

    $ chmod 600 <key_name>.pem

    $ chmod 600 <key_name>.pem

  2. To verify that Vault has been installed, run vault status command and notice that Initialized is false.

    $ vault status

Key                      Value
---                      -----
Recovery Seal Type       awskms
Initialized              false
Sealed                   true
Total Recovery Shares    0
Threshold                0
Unseal Progress          0/0
Unseal Nonce             n/a
Version                  1.9.2
Storage Type             raft
HA Enabled               true

    $ vault status

Key                      Value
---                      -----
Recovery Seal Type       awskms
Initialized              false
Sealed                   true
Total Recovery Shares    0
Threshold                0
Unseal Progress          0/0
Unseal Nonce             n/a
Version                  1.9.2
Storage Type             raft
HA Enabled               true

  3. Run the vault operator init command to initialize the Vault server:

    $ vault operator init

...

Initial Root Token: s.20JnHBY66EKTj9zyR6SjTMNq

Success! Vault is initialized
...

    $ vault operator init

...

Initial Root Token: s.20JnHBY66EKTj9zyR6SjTMNq

Success! Vault is initialized
...

    NOTE: The Vault server is configured to auto-unseal with AWS Key Management Service (KMS); therefore, once the server is initialized, it gets automatically unsealed. To learn how to auto-unseal Vault using AWS KMS, refer to the Auto-unseal using AWS KMS tutorial.

  4. Copy the Initial Root Token value.

  5. Log into Vault. Enter the generated initial root token when prompted.

    $ vault login
Token (will be hidden):

    $ vault login
Token (will be hidden):

  6. Examine and then execute the /home/ubuntu/aws_auth.sh script.

    $ cat aws_auth.sh

    $ cat aws_auth.sh

    The script enables key/value v1 secrets engine at secret and writes some test data at secret/myapp/config (at line 1 and 2).

    At line 4 through 6, it creates a myapp policy, and line 8 enables the aws auth method.

    Finally, it create a role named dev-role-iam which is valid for 24 hours with capabilities set by the myapp policy. (Note: The ${account_id} should be your AWS account ID.) The bound_iam_principal_arn is the Amazon Resource Name (ARN) of the IAM role created by the iam.tf file (at line 11).

    vault secrets enable -path="secret" kv
vault kv put secret/myapp/config ttl='30s' username='appuser' password='suP3rsec(et!'

echo "path \"secret/myapp/*\" {
    capabilities = [\"read\", \"list\"]
}" | vault policy write myapp -

vault auth enable aws
vault write -force auth/aws/config/client

vault write auth/aws/role/dev-role-iam auth_type=iam bound_iam_principal_arn="arn:aws:iam::${account_id}:role/vault-agent-demo-vault-client-role" policies=myapp ttl=24h
    aws_auth.sh
    1 2 3 4 5 6 7 8 9 1011vault secrets enable -path="secret" kv
vault kv put secret/myapp/config ttl='30s' username='appuser' password='suP3rsec(et!'

echo "path \"secret/myapp/*\" {
    capabilities = [\"read\", \"list\"]
}" | vault policy write myapp -

vault auth enable aws
vault write -force auth/aws/config/client

vault write auth/aws/role/dev-role-iam auth_type=iam bound_iam_principal_arn="arn:aws:iam::${account_id}:role/vault-agent-demo-vault-client-role" policies=myapp ttl=24h

  7. Execute the script.

    $ ./aws_auth.sh

Success! Enabled the kv secrets engine at: secret/
Success! Data written to: secret/myapp/config
Success! Uploaded policy: myapp
Success! Enabled aws auth method at: aws/
Success! Data written to: auth/aws/config/client
Success! Data written to: auth/aws/role/dev-role-iam

    $ ./aws_auth.sh

Success! Enabled the kv secrets engine at: secret/
Success! Data written to: secret/myapp/config
Success! Uploaded policy: myapp
Success! Enabled aws auth method at: aws/
Success! Data written to: auth/aws/config/client
Success! Data written to: auth/aws/role/dev-role-iam

  8. Check the myapp policy.

    $ vault policy read myapp

path "secret/myapp/*" {
    capabilities = ["read", "list"]
}

    $ vault policy read myapp

path "secret/myapp/*" {
    capabilities = ["read", "list"]
}

  9. Check the secrets written in secret/myapp/config.

    $ vault kv get secret/myapp/config

====== Data ======
Key         Value
---         -----
password    suP3rsec(et!
ttl         30s
username    appuser

    $ vault kv get secret/myapp/config

====== Data ======
Key         Value
---         -----
password    suP3rsec(et!
ttl         30s
username    appuser

»Run Vault Agent with Auto-Auth

This step should be performed on the Vault Client instance.

Now, you are going to see how Vault Agent Auto-Auth method works, and write out a token to an arbitrary location on disk. Vault Agent is a client daemon and its Auto-Auth feature allows for easy authentication to Vault.

Vault Agent

  1. Open a new terminal and SSH into the Vault Client instance. Output

    When you are prompted, enter "yes" to continue.

    $ ssh -i <path_to_key> ubuntu@<public_ip_of_client>
...
Are you sure you want to continue connecting (yes/no)? yes

    $ ssh -i <path_to_key> ubuntu@<public_ip_of_client>
...
Are you sure you want to continue connecting (yes/no)? yes

  2. In the client instance, explore the Vault Agent configuration file (/home/ubuntu/vault-agent.hcl).

    $ cat vault-agent.hcl

    $ cat vault-agent.hcl

    Starting at line 4, the auto_auth block has two configuration entries: method and sink. In this example, the Auto-Auth is configured to use the aws auth method enabled at the auth/aws path on the Vault server. The Vault Agent will use the dev-role-iam role to authenticate.

    The sink block specifies the location on disk where to write tokens. Vault Agent Auto-Auth sink can be configured multiple times if you want Vault Agent to place the token into multiple locations. In this example, the sink is set to /home/ubuntu/vault-token-via-agent.

    The exit_after_auth parameter is set to true; therefore, the agent will exit with code 0 after a single successful authentication. The default is false and the agent continues to run and automatically renew the client token for you.

    exit_after_auth = true
pid_file = "./pidfile"

auto_auth {
  method "aws" {
      mount_path = "auth/aws"
      config = {
          type = "iam"
          role = "dev-role-iam"
      }
  }

  sink "file" {
      config = {
          path = "/home/ubuntu/vault-token-via-agent"
      }
  }
}

vault {
  address = "http://10.0.101.53:8200"
}
    vault-agent.hcl
    1 2 3 4 5 6 7 8 9 10111213141516171819202122exit_after_auth = true
pid_file = "./pidfile"

auto_auth {
  method "aws" {
      mount_path = "auth/aws"
      config = {
          type = "iam"
          role = "dev-role-iam"
      }
  }

  sink "file" {
      config = {
          path = "/home/ubuntu/vault-token-via-agent"
      }
  }
}

vault {
  address = "http://10.0.101.53:8200"
}

    The vault block points to the Vault server address. This should match to the private IP address of your Vault server host.

    NOTE: For the full details of Vault Agent configuration parameters, refer to the Vault Agent documentation.

  3. Execute the following command to get help:

    $ vault agent -h

    $ vault agent -h

  4. Now, you are ready to run the Vault Agent. Execute the following command:

    $ vault agent -config=/home/ubuntu/vault-agent.hcl -log-level=debug
==> Vault agent started! Log data will stream in below:

==> Vault agent configuration:

                     Cgo: disabled
               Log Level: debug
                 Version: Vault v1.9.2
             Version Sha: f4c6d873e2767c0d6853b5d9ffc77b0d297bfbdf

2022-01-25T15:32:58.457Z [INFO]  sink.file: creating file sink
2022-01-25T15:32:58.457Z [INFO]  sink.file: file sink configured: path=/home/ubuntu/vault-token-via-agent mode=-rw-r-----
2022-01-25T15:32:58.462Z [INFO]  template.server: starting template server
2022-01-25T15:32:58.462Z [INFO]  template.server: no templates found
2022-01-25T15:32:58.462Z [INFO]  auth.handler: starting auth handler
2022-01-25T15:32:58.462Z [INFO]  auth.handler: authenticating
2022-01-25T15:32:58.463Z [INFO]  sink.server: starting sink server
2022-01-25T15:32:58.511Z [INFO]  auth.handler: authentication successful, sending token to sinks
2022-01-25T15:32:58.511Z [INFO]  auth.handler: starting renewal process
2022-01-25T15:32:58.511Z [INFO]  sink.file: token written: path=/home/ubuntu/vault-token-via-agent
2022-01-25T15:32:58.511Z [INFO]  sink.server: sink server stopped
2022-01-25T15:32:58.512Z [INFO]  sinks finished, exiting
2022-01-25T15:32:58.512Z [INFO]  template.server: template server stopped
2022-01-25T15:32:58.512Z [INFO]  auth.handler: shutdown triggered, stopping lifetime watcher
2022-01-25T15:32:58.512Z [INFO]  auth.handler: auth handler stopped

    $ vault agent -config=/home/ubuntu/vault-agent.hcl -log-level=debug
==> Vault agent started! Log data will stream in below:

==> Vault agent configuration:

                     Cgo: disabled
               Log Level: debug
                 Version: Vault v1.9.2
             Version Sha: f4c6d873e2767c0d6853b5d9ffc77b0d297bfbdf

2022-01-25T15:32:58.457Z [INFO]  sink.file: creating file sink
2022-01-25T15:32:58.457Z [INFO]  sink.file: file sink configured: path=/home/ubuntu/vault-token-via-agent mode=-rw-r-----
2022-01-25T15:32:58.462Z [INFO]  template.server: starting template server
2022-01-25T15:32:58.462Z [INFO]  template.server: no templates found
2022-01-25T15:32:58.462Z [INFO]  auth.handler: starting auth handler
2022-01-25T15:32:58.462Z [INFO]  auth.handler: authenticating
2022-01-25T15:32:58.463Z [INFO]  sink.server: starting sink server
2022-01-25T15:32:58.511Z [INFO]  auth.handler: authentication successful, sending token to sinks
2022-01-25T15:32:58.511Z [INFO]  auth.handler: starting renewal process
2022-01-25T15:32:58.511Z [INFO]  sink.file: token written: path=/home/ubuntu/vault-token-via-agent
2022-01-25T15:32:58.511Z [INFO]  sink.server: sink server stopped
2022-01-25T15:32:58.512Z [INFO]  sinks finished, exiting
2022-01-25T15:32:58.512Z [INFO]  template.server: template server stopped
2022-01-25T15:32:58.512Z [INFO]  auth.handler: shutdown triggered, stopping lifetime watcher
2022-01-25T15:32:58.512Z [INFO]  auth.handler: auth handler stopped

    NOTE: Because the vault-agent.hcl configuration file contained the line exit_after_auth = true, Vault Agent simply authenticated and retrieved a token once, wrote it to the defined sink, and exited. Vault Agent can also run in daemon mode where it will continuously renew the retrieved token, and attempt to re-authenticate if that token becomes invalid. Vault Agent Caching tutorial will demonstrate running Vault Agent as a daemon.

  5. The returned token is written to /home/ubuntu/vault-token-via-agent.

    $ more vault-token-via-agent
s.6Wqj0YNiRngCCf6Cic3dA5Ul

    $ more vault-token-via-agent
s.6Wqj0YNiRngCCf6Cic3dA5Ul

  6. Let's try an API call using the token that Vault Agent pulled for us to test:

    $ curl --header "X-Vault-Token: $(cat /home/ubuntu/vault-token-via-agent)" \
      $VAULT_ADDR/v1/secret/myapp/config | jq -r ".data"

    $ curl --header "X-Vault-Token: $(cat /home/ubuntu/vault-token-via-agent)" \
      $VAULT_ADDR/v1/secret/myapp/config | jq -r ".data"

    Output:

    {
  "password": "suP3rsec(et!",
  "ttl": "30s",
  "username": "appuser"
}

    {
  "password": "suP3rsec(et!",
  "ttl": "30s",
  "username": "appuser"
}

»Response Wrap the token

This step should be performed on the Vault Client instance.

It may not be ideal to write the token as a plaintext depending on the firewall around the client instance. Vault Agent supports response-wrapping of the token to provide an additional layer of protection for the token. Tokens can be wrapped by either the auth method or by the sink configuration, with each approach solving for different challenges as described in Response-Wrapping Tokens.

  1. In the client instance, explore the /home/ubuntu/vault-agent-wrapped.hcl file which supports response-wrapping of the token.

    $ cat /home/ubuntu/vault-agent-wrapped.hcl

exit_after_auth = true
pid_file = "./pidfile"

auto_auth {
    method "aws" {
        mount_path = "auth/aws"
        config = {
            type = "iam"
            role = "dev-role-iam"
        }
    }

    sink "file" {
        wrap_ttl = "5m"
        config = {
            path = "/home/ubuntu/vault-token-via-agent"
        }
    }
}

vault {
  address = "http://10.0.101.53:8200"
}

    $ cat /home/ubuntu/vault-agent-wrapped.hcl

exit_after_auth = true
pid_file = "./pidfile"

auto_auth {
    method "aws" {
        mount_path = "auth/aws"
        config = {
            type = "iam"
            role = "dev-role-iam"
        }
    }

    sink "file" {
        wrap_ttl = "5m"
        config = {
            path = "/home/ubuntu/vault-token-via-agent"
        }
    }
}

vault {
  address = "http://10.0.101.53:8200"
}

    Notice the wrap_ttl parameter in the sink block. This configuration uses the sink method to response-wrap the retrieved tokens.

  2. From the client, run the Vault Agent again and inspect the output:

    $ vault agent -config=/home/ubuntu/vault-agent-wrapped.hcl -log-level=debug

==> Vault agent started! Log data will stream in below:

==> Vault agent configuration:

                     Cgo: disabled
               Log Level: debug
                 Version: Vault v1.9.2
             Version Sha: f4c6d873e2767c0d6853b5d9ffc77b0d297bfbdf

2022-01-25T15:32:14.659Z [INFO]  sink.file: creating file sink
2022-01-25T15:32:14.659Z [INFO]  sink.file: file sink configured: path=/home/ubuntu/vault-token-via-agent mode=-rw-r-----
2022-01-25T15:32:14.667Z [INFO]  template.server: starting template server
2022-01-25T15:32:14.667Z [INFO]  template.server: no templates found
2022-01-25T15:32:14.667Z [INFO]  auth.handler: starting auth handler
2022-01-25T15:32:14.667Z [INFO]  auth.handler: authenticating
2022-01-25T15:32:14.668Z [INFO]  sink.server: starting sink server
2022-01-25T15:32:14.707Z [INFO]  auth.handler: authentication successful, sending token to sinks
2022-01-25T15:32:14.707Z [INFO]  auth.handler: starting renewal process
2022-01-25T15:32:14.723Z [INFO]  auth.handler: renewed auth token
2022-01-25T15:32:14.739Z [INFO]  sink.file: token written: path=/home/ubuntu/vault-token-via-agent
2022-01-25T15:32:14.740Z [INFO]  sink.server: sink server stopped
2022-01-25T15:32:14.740Z [INFO]  sinks finished, exiting
2022-01-25T15:32:14.740Z [INFO]  template.server: template server stopped
2022-01-25T15:32:14.740Z [INFO]  auth.handler: shutdown triggered, stopping lifetime watcher
2022-01-25T15:32:14.740Z [INFO]  auth.handler: auth handler stopped

    $ vault agent -config=/home/ubuntu/vault-agent-wrapped.hcl -log-level=debug

==> Vault agent started! Log data will stream in below:

==> Vault agent configuration:

                     Cgo: disabled
               Log Level: debug
                 Version: Vault v1.9.2
             Version Sha: f4c6d873e2767c0d6853b5d9ffc77b0d297bfbdf

2022-01-25T15:32:14.659Z [INFO]  sink.file: creating file sink
2022-01-25T15:32:14.659Z [INFO]  sink.file: file sink configured: path=/home/ubuntu/vault-token-via-agent mode=-rw-r-----
2022-01-25T15:32:14.667Z [INFO]  template.server: starting template server
2022-01-25T15:32:14.667Z [INFO]  template.server: no templates found
2022-01-25T15:32:14.667Z [INFO]  auth.handler: starting auth handler
2022-01-25T15:32:14.667Z [INFO]  auth.handler: authenticating
2022-01-25T15:32:14.668Z [INFO]  sink.server: starting sink server
2022-01-25T15:32:14.707Z [INFO]  auth.handler: authentication successful, sending token to sinks
2022-01-25T15:32:14.707Z [INFO]  auth.handler: starting renewal process
2022-01-25T15:32:14.723Z [INFO]  auth.handler: renewed auth token
2022-01-25T15:32:14.739Z [INFO]  sink.file: token written: path=/home/ubuntu/vault-token-via-agent
2022-01-25T15:32:14.740Z [INFO]  sink.server: sink server stopped
2022-01-25T15:32:14.740Z [INFO]  sinks finished, exiting
2022-01-25T15:32:14.740Z [INFO]  template.server: template server stopped
2022-01-25T15:32:14.740Z [INFO]  auth.handler: shutdown triggered, stopping lifetime watcher
2022-01-25T15:32:14.740Z [INFO]  auth.handler: auth handler stopped

    Instead of a token value, you now have a JSON object containing a wrapping token as well as some additional metadata. In order to get to the true token, you need to first perform an unwrap operation.

  3. Unwrap the response-wrapped token and save it to a VAULT_TOKEN env var that other applications can use:

    $ export VAULT_TOKEN=$(vault unwrap -field=token $(jq -r '.token' /home/ubuntu/vault-token-via-agent))

    $ export VAULT_TOKEN=$(vault unwrap -field=token $(jq -r '.token' /home/ubuntu/vault-token-via-agent))

  4. View the unwrapped token value.

    $ echo $VAULT_TOKEN
s.3VaVYxv0rNgb5JLZBQqS8BXX

    $ echo $VAULT_TOKEN
s.3VaVYxv0rNgb5JLZBQqS8BXX

  5. Test to make sure that the token has the read permission on secret/myapp/config.

    $ curl --header "X-Vault-Token: $VAULT_TOKEN" $VAULT_ADDR/v1/secret/myapp/config \
   | jq -r ".data"

    $ curl --header "X-Vault-Token: $VAULT_TOKEN" $VAULT_ADDR/v1/secret/myapp/config \
   | jq -r ".data"

    Notice that the value saved to the VAULT_TOKEN is not the same as the token value in the /home/ubuntu/vault-token-via-agent file. The value in VAULT_TOKEN is the unwrapped token retrieved by Vault Agent.

  6. If you try to unwrap that same value again or wait longer than 5 minutes (wrap_ttl), it will throw an error:

    $ export VAULT_TOKEN=$(vault unwrap -field=token $(jq -r '.token' /home/ubuntu/vault-token-via-agent))

Error unwrapping: Error making API request.

URL: PUT http://active.vault-va-demo.service.dc1.consul:8200/v1/sys/wrapping/unwrap
Code: 400. Errors:

* wrapping token is not valid or does not exist

    $ export VAULT_TOKEN=$(vault unwrap -field=token $(jq -r '.token' /home/ubuntu/vault-token-via-agent))

Error unwrapping: Error making API request.

URL: PUT http://active.vault-va-demo.service.dc1.consul:8200/v1/sys/wrapping/unwrap
Code: 400. Errors:

* wrapping token is not valid or does not exist

    A response-wrapped token can only be unwrapped once. Additional attempts to unwrap an already-unwrapped token will result in triggering an error.

    NOTE: In this tutorial, the basic mechanics of Vault Agent was demonstrated. Read the Additional Discussion section about the next step.

»Clean up

Return to the first terminal where you created the cluster and use Terraform to destroy the cluster.

Destroy the AWS resources provisioned by Terraform.

$ terraform destroy -auto-approve

$ terraform destroy -auto-approve

Delete the state file.

$ rm *tfstate*

$ rm *tfstate*

»Additional Discussion

In the above examples, you manually ran Vault Agent in order to demonstrate how it works. How you actually integrate Vault Agent into your application deployment workflow will vary with a number of factors. Some questions to ask to help determine appropriate usage:

  • What is the lifecycle of my application? Is it more ephemeral or long-lived?
  • What are the lifecycles of my authentication tokens? Are they long-lived and simply need to be renewed over and over to demonstrate liveliness of a service or do you want to enforce periodic re-authentications?
  • Do I have a number of applications running on my host that each needs their own token? Can I use a native authentication capability (e.g. AWS IAM, K8s, Azure MSI, Google Cloud IAM, etc.)?

The answers to these questions will help you determine if Vault Agent should run as a daemon or as a prerequisite of a service configuration. Take for example the following Systemd service definition for running Nomad:

[Unit]
Description=Nomad Agent
Requires=consul-online.target
After=consul-online.target

[Service]
KillMode=process
KillSignal=SIGINT
Environment=VAULT_ADDR=http://active.vault.service.consul:8200
Environment=VAULT_SKIP_VERIFY=true
ExecStartPre=/usr/local/bin/vault agent -config /etc/vault-agent.d/vault-agent.hcl
ExecStart=/usr/bin/nomad-vault.sh
ExecReload=/bin/kill -HUP $MAINPID
Restart=on-failure
RestartSec=2
StartLimitBurst=3
StartLimitIntervalSec=10
LimitNOFILE=65536

[Install]
WantedBy=multi-user.target

[Unit]
Description=Nomad Agent
Requires=consul-online.target
After=consul-online.target

[Service]
KillMode=process
KillSignal=SIGINT
Environment=VAULT_ADDR=http://active.vault.service.consul:8200
Environment=VAULT_SKIP_VERIFY=true
ExecStartPre=/usr/local/bin/vault agent -config /etc/vault-agent.d/vault-agent.hcl
ExecStart=/usr/bin/nomad-vault.sh
ExecReload=/bin/kill -HUP $MAINPID
Restart=on-failure
RestartSec=2
StartLimitBurst=3
StartLimitIntervalSec=10
LimitNOFILE=65536

[Install]
WantedBy=multi-user.target

Notice the ExecStartPre directive that runs Vault Agent before the desired service starts. The service startup script expects a Vault token to be set as is demonstrated in the /usr/bin/nomad-vault.sh startup script:

#!/usr/bin/env bash

if [ -f /mnt/ramdisk/token ]; then
  exec env VAULT_TOKEN=$(vault unwrap -field=token $(jq -r '.token' /mnt/ramdisk/token)) \
    /usr/local/bin/nomad agent \
      -config=/etc/nomad.d \
      -vault-tls-skip-verify=true
else
  echo "Nomad service failed due to missing Vault token"
  exit 1
fi

#!/usr/bin/env bash

if [ -f /mnt/ramdisk/token ]; then
  exec env VAULT_TOKEN=$(vault unwrap -field=token $(jq -r '.token' /mnt/ramdisk/token)) \
    /usr/local/bin/nomad agent \
      -config=/etc/nomad.d \
      -vault-tls-skip-verify=true
else
  echo "Nomad service failed due to missing Vault token"
  exit 1
fi

Many applications that expect Vault tokens typically look for a VAULT_TOKEN environment variable. Here, you're using Vault Agent to obtain a token and write it out to a RAM disk and as part of the Nomad startup script, you read the response-wrapped token from the RAM disk and save it to our VAULT_TOKEN environment variable before actually starting Nomad.

Furthermore, since Vault Agent Auto-Auth only addresses the challenges of obtaining and managing authentication tokens (secret zero), you might want to use helper tools such as Consul Template and Envconsul to obtain secrets stored in Vault (e.g. DB credentials, PKI certificates, AWS access keys, etc.).

»Help and reference

Other Vault Agent tutorials:

To learn more about the response wrapping feature, refer the following:

stdin: is not a tty