Nearly all requests to Vault must be accompanied by an authentication token. This includes all API requests, as well as via the Vault CLI and other libraries.
Vault provides a number of different authentication methods to assist in delivery of this initial token. If you can securely get the first secret from an originator to a consumer, all subsequent secrets transmitted between this originator and consumer can be authenticated with the trust established by the successful distribution and user of that first secret. Getting the first secret to the consumer, is the secure introduction challenge.
To that end, Vault provides integration with native authentication capabilities in various environments, for example: IAM in AWS and Google Cloud, Managed Service Identities in Azure, and Service Accounts in Kubernetes.
»Challenge
Although a number of auth methods are available, the client is still responsible for managing the lifecycle of its Vault tokens. Therefore, the challenge becomes how to enable authentication to Vault and manage the lifecycle of tokens in a standard way without having to write custom logic.
»Solution
Vault Agent provides a number of different helper features, specifically addressing the following challenges:
- Automatic authentication
- Secure delivery/storage of tokens
- Lifecycle management of these tokens (renewal & re-authentication)
NOTE: The Vault Agent Auto-Auth functionality addresses the challenges related to obtaining and managing authentication tokens (secret zero).
The Secure Introduction of Vault Clients introduced three basic approaches: Platform Integration, Trusted Orchestrator, and Vault Agent. This tutorial demonstrates how Vault Agent works.
»Prerequisites
To complete this section of the tutorial, you will need:
- Terraform v0.15.2 or later installed
- AWS account and associated credentials that allow for the creation of resources.
»Provision the Cloud Resources
Clone the demo assets from the hashicorp/vault-guides GitHub repository to perform the steps described in this tutorial.
This repository contains supporting content for all of the Vault learn tutorials. The content specific to this tutorial can be found within a sub-directory.
Be sure to set your working directory to where the
/identity/vault-agent-demo/terraform-aws
folder is located.The working directory should contain the provided Terraform files:
NOTE: The example Terraform in this repository is created for the demo purpose, and not suitable for production use. For production deployment, refer the example Terraform in the /operations/provision-vault folder.
Set an
AWS_ACCESS_KEY_ID
environment variable to hold your AWS access key ID.Set an
AWS_SECRET_ACCESS_KEY
environment variable to hold your AWS secret access key.Tip: The above example uses IAM user authentication. You can use any authentication method described in the AWS provider documentation.
Create a file named
terraform.tfvars
and specify thekey_name
. (You can copy theterraform.tfvars.example
file and rename it asterraform.tfvars
.)Example:
terraform.tfvarsIf you don't have an EC2 key pair, follow the AWS documentation to create one.
Perform a
terraform init
to pull down the necessary provider resources.Run
terraform apply
and review the planned actions. Your terminal output should indicate the plan is running and what resources will be created.Enter
yes
to confirm and resume.When the
apply
command completes, the Terraform output will display the public IP address to SSH into your Vault server and client instances.Example output:
»Configure AWS IAM Auth Method
This step should be performed on the Vault Server instance.
In this step, you will configure Vault to allow AWS IAM authentication from specific IAM roles.
SSH into the Vault Server instance.
When you are prompted, enter "yes" to continue.
NOTE: If you received
Permissions 0664 for '<key_name>.pem' are too open
error, be sure to set the file permission appropriately.To verify that Vault has been installed, run
vault status
command and notice that Initialized isfalse
.Run the
vault operator init
command to initialize the Vault server:NOTE: The Vault server is configured to auto-unseal with AWS Key Management Service (KMS); therefore, once the server is initialized, it gets automatically unsealed. To learn how to auto-unseal Vault using AWS KMS, refer to the Auto-unseal using AWS KMS tutorial.
Copy the Initial Root Token value.
Log into Vault. Enter the generated initial root token when prompted.
Examine and then execute the
/home/ubuntu/aws_auth.sh
script.The script enables key/value v1 secrets engine at
secret
and writes some test data atsecret/myapp/config
(at line 1 and 2).At line 4 through 6, it creates a
myapp
policy, and line 8 enables theaws
auth method.Finally, it create a role named
dev-role-iam
which is valid for 24 hours with capabilities set by themyapp
policy. (Note: The${account_id}
should be your AWS account ID.) Thebound_iam_principal_arn
is the Amazon Resource Name (ARN) of the IAM role created by theiam.tf
file (at line 11).aws_auth.sh1 2 3 4 5 6 7 8 9 1011
Execute the script.
Check the
myapp
policy.Check the secrets written in
secret/myapp/config
.
»Run Vault Agent with Auto-Auth
This step should be performed on the Vault Client instance.
Now, you are going to see how Vault Agent Auto-Auth method works, and write out a token to an arbitrary location on disk. Vault Agent is a client daemon and its Auto-Auth feature allows for easy authentication to Vault.
Open a new terminal and SSH into the Vault Client instance.
When you are prompted, enter "yes" to continue.
In the client instance, explore the Vault Agent configuration file (
/home/ubuntu/vault-agent.hcl
).Starting at line 4, the
auto_auth
block has two configuration entries:method
andsink
. In this example, the Auto-Auth is configured to use theaws
auth method enabled at theauth/aws
path on the Vault server. The Vault Agent will use thedev-role-iam
role to authenticate.The
sink
block specifies the location on disk where to write tokens. Vault Agent Auto-Authsink
can be configured multiple times if you want Vault Agent to place the token into multiple locations. In this example, thesink
is set to/home/ubuntu/vault-token-via-agent
.The
exit_after_auth
parameter is set totrue
; therefore, the agent will exit with code 0 after a single successful authentication. The default isfalse
and the agent continues to run and automatically renew the client token for you.vault-agent.hcl1 2 3 4 5 6 7 8 9 10111213141516171819202122
The
vault
block points to the Vault serveraddress
. This should match to the private IP address of your Vault server host.NOTE: For the full details of Vault Agent configuration parameters, refer to the Vault Agent documentation.
Execute the following command to get help:
Now, you are ready to run the Vault Agent. Execute the following command:
NOTE: Because the
vault-agent.hcl
configuration file contained the lineexit_after_auth = true
, Vault Agent simply authenticated and retrieved a token once, wrote it to the defined sink, and exited. Vault Agent can also run in daemon mode where it will continuously renew the retrieved token, and attempt to re-authenticate if that token becomes invalid. Vault Agent Caching tutorial will demonstrate running Vault Agent as a daemon.The returned token is written to
/home/ubuntu/vault-token-via-agent
.Let's try an API call using the token that Vault Agent pulled for us to test:
Output:
»Response Wrap the token
This step should be performed on the Vault Client instance.
It may not be ideal to write the token as a plaintext depending on the firewall around the client instance. Vault Agent supports response-wrapping of the token to provide an additional layer of protection for the token. Tokens can be wrapped by either the auth method or by the sink configuration, with each approach solving for different challenges as described in Response-Wrapping Tokens.
In the client instance, explore the
/home/ubuntu/vault-agent-wrapped.hcl
file which supports response-wrapping of the token.Notice the
wrap_ttl
parameter in thesink
block. This configuration uses the sink method to response-wrap the retrieved tokens.From the client, run the Vault Agent again and inspect the output:
Instead of a token value, you now have a JSON object containing a wrapping token as well as some additional metadata. In order to get to the true token, you need to first perform an
unwrap
operation.Unwrap the response-wrapped token and save it to a
VAULT_TOKEN
env var that other applications can use:View the unwrapped token value.
Test to make sure that the token has the read permission on
secret/myapp/config
.Notice that the value saved to the
VAULT_TOKEN
is not the same as thetoken
value in the/home/ubuntu/vault-token-via-agent
file. The value inVAULT_TOKEN
is the unwrapped token retrieved by Vault Agent.If you try to unwrap that same value again or wait longer than 5 minutes (
wrap_ttl
), it will throw an error:A response-wrapped token can only be unwrapped once. Additional attempts to unwrap an already-unwrapped token will result in triggering an error.
NOTE: In this tutorial, the basic mechanics of Vault Agent was demonstrated. Read the Additional Discussion section about the next step.
»Clean up
Return to the first terminal where you created the cluster and use Terraform to destroy the cluster.
Destroy the AWS resources provisioned by Terraform.
Delete the state file.
»Additional Discussion
In the above examples, you manually ran Vault Agent in order to demonstrate how it works. How you actually integrate Vault Agent into your application deployment workflow will vary with a number of factors. Some questions to ask to help determine appropriate usage:
- What is the lifecycle of my application? Is it more ephemeral or long-lived?
- What are the lifecycles of my authentication tokens? Are they long-lived and simply need to be renewed over and over to demonstrate liveliness of a service or do you want to enforce periodic re-authentications?
- Do I have a number of applications running on my host that each needs their own token? Can I use a native authentication capability (e.g. AWS IAM, K8s, Azure MSI, Google Cloud IAM, etc.)?
The answers to these questions will help you determine if Vault Agent should run
as a daemon or as a prerequisite of a service configuration. Take for example
the following Systemd
service definition for running Nomad:
Notice the ExecStartPre
directive that runs Vault Agent before the desired
service starts. The service startup script expects a Vault token to be set as is
demonstrated in the /usr/bin/nomad-vault.sh
startup script:
Many applications that expect Vault tokens typically look for a VAULT_TOKEN
environment variable. Here, you're using Vault Agent to obtain a token and write it out to a
RAM disk and as part of the Nomad startup script, you read the response-wrapped
token from the RAM disk and save it to our VAULT_TOKEN
environment variable before actually
starting Nomad.
Furthermore, since Vault Agent Auto-Auth only addresses the challenges of obtaining and managing authentication tokens (secret zero), you might want to use helper tools such as Consul Template and Envconsul to obtain secrets stored in Vault (e.g. DB credentials, PKI certificates, AWS access keys, etc.).
»Help and reference
- Blog post: Why Use the Vault Agent for Secrets Management?
- Video: Streamline Secrets Management with Vault Agent and Vault 0.11
- Secure Introduction of Vault Clients
- Vault Agent Auto-Auth documentation
- AWS Auth Method documentation
Other Vault Agent tutorials:
- Vault Agent with Kubernetes
- Vault Agent Templates
- Vault Agent Caching
- Vault Agent Windows Service
- Read Secrets From Vault Using Vault Agent
- Using HashiCorp Vault Agent with .NET Core
To learn more about the response wrapping feature, refer the following: