Virtual Event
Join us for the next HashiConf Digital October 12-15, 2020 Register for Free

Inject Secrets with Vault

Inject secrets into Terraform using the Vault provider

Traditionally, developers looking to safely provision infrastructure using Terraform are given their own set of long-lived, scoped AWS credentials. While this enables the developer's freedom, using long-lived credentials can be dangerous and difficult to secure.

  1. Operators need to manage a large number of static, long-lived AWS IAM credentials with varying scope.

  2. Long-lived credentials on a developer's local machine creates a large attack surface area. If a malicious actor gains access to the credentials, they could used them to damage resources.

You can address both concerns by storing your long-lived AWS credentials in HashiCorp's Vault's AWS Secrets Engine, then leverage Terraform's Vault provider to generate appropriately scoped & short-lived AWS credentials to be used by Terraform to provision resources in AWS.

As a result, operators (Vault Admin) are able to avoid managing static, long-lived secrets with varying scope and developers (Terraform Operator) are able to provision resources without having direct access to the secrets.

In this guide, you assume the role of both the Vault Admin and the Terraform Operator.

AWS secret injection in Terraform with Vault requires both a Vault Admin (to configure and manage the secrets) and a Terraform Operator (to use the secrets).

  1. First, as an Vault Admin, you will configure AWS Secrets Engine in Vault.

  2. Then, as a Terraform Operator, you will use connect to the Vault instance to retrieve dynamic, short-lived AWS credentials generated by the AWS Secrets Engine to provision an Ubuntu EC2 instance.

  3. Finally, as an Vault Admin, you will remove the Terraform Operator's ability to manipulate EC2 instances by modifying the policy for the corresponding Vault role.

Throughout this journey, you'll learn about the benefits and considerations this approach has to offer.

»Prerequisites

In order to follow this guide, you should be familiar with the usual Terraform plan/apply workflow and Vault. If you're new to Terraform, refer first to the Terraform Getting Started guide. If you're new to Vault, refer first to the Vault Getting Started guide.

In addition, you will need the following:

  • Terraform installed locally

  • Vault installed locally

  • an AWS account and AWS Access Credentials

    If you don't have AWS Access Credentials, create your AWS Access Key ID and Secret Access Key by navigating to your service credentials in the IAM service on AWS. Click "Create access key" here and download the file. This file contains your access credentials.

»Start Vault server

Start a Vault server in development mode with education as the root token. Leave this process running in your terminal window.

$ vault server -dev -dev-root-token-id="education"
==> Vault server configuration:

    Api Address: http://127.0.0.1:8200
            Cgo: disabled
Cluster Address: https://127.0.0.1:8201
     Go Version: go1.14.4
     Listener 1: tcp (addr: "127.0.0.1:8200", cluster address: "127.0.0.1:8201", max_request_duration: "1m30s", max_request_size: "33554432", tls: "disabled")
      Log Level: info
          Mlock: supported: false, enabled: false
  Recovery Mode: false
        Storage: inmem
        Version: Vault v1.4.3
    Version Sha: 491533b63ec9c1343eac3a24d8a7558185a0acb7+CHANGES

Your Vault server should now be up. Navigate to localhost:8200 and login into the instance using your root token: education.

Login to local Vault instance using root token: education.

»Clone repository

In your terminal, clone the Inject Secrets repository and navigate into the directory. It contains the example configurations used in this guide.

$ git clone https://github.com/hashicorp/learn-terraform-inject-secrets-aws-vault && cd learn-terraform-inject-secrets-aws-vault

This directory should contain two Terraform workspaces — an vault-admin-workspace and a operator-workspace.

$ tree
.
├── README.md
├── operator-workspace
│   └── main.tf
└── vault-admin-workspace
    └── main.tf

»Configure AWS Secrets Engine in Vault

In another terminal window (leave the Vault instance running), navigate to the Vault Admin directory.

$ cd vault-admin-workspace

In the main.tf file, you will find 2 resources:

  1. the vault_aws_secret_backend.aws resource configures AWS Secrets Engine to generate a dynamic token that lasts for 2 minutes.

  2. the vault_aws_secret_backend_role.admin resource configures a role for the AWS Secrets Engine named dynamic-aws-creds-vault-admin-role with an IAM policy that allows it iam:* and ec2:* permissions.

    This role will be used by the Terraform Operator workspace to dynamically generate AWS credentials scoped to this IAM policy.

Before applying this configuration, set the required Terraform variable substituting <AWS_ACCESS_KEY_ID> and <AWS_SECRET_ACCESS_KEY> with your AWS Credentials. Notice that we're also setting the required Vault Provider arguments as environment variables: VAULT_ADDR & VAULT_TOKEN.

$ export TF_VAR_aws_access_key=<AWS_ACCESS_KEY_ID>
$ export TF_VAR_aws_secret_key=<AWS_SECRET_ACCESS_KEY>
$ export VAULT_ADDR=http://127.0.0.1:8200
$ export VAULT_TOKEN=education

Initialize the Vault Admin workspace.

$ terraform init

In your initialized directory, run terraform apply, review the planned actions, and confirm the run with a yes

$ terraform apply

## ... Output truncated ...

Apply complete! Resources: 2 added, 0 changed, 0 destroyed.

The state of your infrastructure has been saved to the path
below. This state is required to modify and destroy your
infrastructure, so keep it safe. To inspect the complete state
use the `terraform show` command.

State path: terraform.tfstate

Outputs:

backend = dynamic-aws-creds-vault-admin-path
role = dynamic-aws-creds-vault-admin-role

Notice that there are two output variables named backend and role. These output variables will be used by the Terraform Operator workspace in a later step.

If you go to the terminal where your Vault server is running, you should see Vault output something similar to the below. This means Terraform was successfully able to mount the AWS Secrets Engine at the specified path. The role has also been configured although it's not output in the logs.

2020-07-13T14:01:26.164-0700 [INFO]  core: successful mount: namespace= path=dynamic-aws-creds-vault-admin-path/ type=aws

»Provision compute instance

Now that you have successfully configured Vault's AWS Secrets Engine, you can retrieve dynamic short lived AWS token to provision an EC2 instance.

Navigate to the Terraform Operator workspace.

$ cd ../operator-workspace

In the main.tf file, you should find the following data and resource blocks:

  1. the terraform_remote_state.admin data block retrieves the Terraform state file generated from your Vault Admin workspace

  2. the vault_aws_access_credentials.creds data block retrieves the dynamic, short-lived AWS credentials from your Vault instance. Notice that this uses the Vault Admin workspace's output variables: backend and role

  3. the aws provider is initialized with the short-lived credentials retrieved by vault_aws_access_credentials.creds. The provider is configured to the us-east-1 region, as defined by the region variable

  4. the aws_ami.ubuntu data block retrieves the most recent Ubuntu image

  5. the aws_instance.main resource block creates an t2.micro EC2 instance

Initialize the Terraform Operator workspace.

$ terraform init

Navigate to the IAM Users page in AWS Console. Search for the username prefix vault-token-terraform-dynamic-aws-creds-vault-admin. Nothing should show up on your initial search. However, a user with this prefix should appear on terraform plan or terraform apply.

Apply the Terraform configuration, remember to confirm the run with a yes. Terraform will provision the EC2 instance using the dynamic credentials generated from Vault.

$ terraform apply

Refresh the IAM Users and search for the vault-token-terraform-dynamic-aws-creds-vault-admin prefix. You should see a IAM user.

Dynamic, short-lived IAM user in AWS Console

This IAM user was generated by Vault with the appropriate IAM policy configured by the Vault Admin workspace. Because the default_lease_ttl_seconds is set to 120 seconds, Vault will revoke those IAM credentials and they will be removed from the AWS IAM console after 120 seconds.

Navigate to the EC2 page and search for dynamic-aws-creds-operator. You should see an instance provisioned by the Terraform Operator workspace using the short-lived AWS credentials.

AWS Console showing EC2 instance provisioned by Terraform Operator workspace

Every Terraform run with this configuration will use its own unique set of AWS IAM credentials that are scoped to whatever the Vault Admin has defined.

The Terraform Operator doesn't have to manage long-lived AWS credentials locally. The Vault Admin only has to manage the Vault role rather than numerous, multi-scoped, long-lived AWS credentials.

After 120 seconds, you should see the following in the terminal running Vault.

2020-07-13T16:07:55.755-0700 [INFO]  expiration: revoked lease: lease_id=dynamic-aws-creds-vault-admin-path/creds/dynamic-aws-creds-vault-admin-role/z1PKR7Y623fk0ZQWW1kwaVVY

This shows that Vault has destroyed the short-lived AWS credentials generated for the apply run.

»Destroy EC2 instance

Destroy the EC2 instance, remember to confirm the run with a yes.

$ terraform destroy

This run should have generated and used another set of IAM credentials. Verify that your EC2 instance has been destroyed by viewing the EC2 page of your AWS Console.

»Restrict Vault role's permissions

If the Vault Admin wanted to remove the Terraform Operator's EC2 permissions, they would only need to update the Vault role's policy.

Navigate to the Vault Admin workspace.

$ cd ../vault-admin-workspace

Remove "ec2:*" from the vault_aws_secret_backend_role.admin resource in your main.tf file.

$ sed -i '' -e 's/, \"ec2:\*\"//g' main.tf
resource "vault_aws_secret_backend_role" "admin" {
  backend = vault_aws_secret_backend.aws.path
  name    = "${var.name}-role"
  credential_type = "iam_user"

  policy_document = <<EOF
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
-        "iam:*", "ec2:*"
+        "iam:*"
      ],
      "Resource": "*"
    }
  ]
}
EOF
}

This change restricts the Terraform Operator's ability to provision any AWS EC2 instance.

Apply the Terraform configuration, remember to confirm the run with a yes.

$ terraform apply

»Verify restricted Terraform Operator permissions

Navigate to the Terraform Operator workspace.

$ cd ../operator-workspace

Run terraform plan. This plan should fail because the Terraform Operator no longer has CRUD permissions on EC2 instances due to changes to the dynamic-aws-creds-vault-admin role.

$ terraform plan
Refreshing Terraform state in-memory prior to plan...
The refreshed state will be used to calculate this plan, but will not be
persisted to local or remote state storage.

data.terraform_remote_state.admin: Refreshing state...
data.vault_aws_access_credentials.creds: Refreshing state...
data.aws_ami.ubuntu: Refreshing state...

Error: UnauthorizedOperation: You are not authorized to perform this operation.
        status code: 403, request id: 5740c8fd-0bfd-4bf7-94c3-d56a58ae7f39

  on main.tf line 31, in data "aws_ami" "ubuntu":
  31: data "aws_ami" "ubuntu" {

»Benefits and considerations

This approach to secret injection:

  1. alleviates the Vault Admin's responsibility in managing numerous, multi-scoped, long-lived AWS credentials,

  2. reduces the risk from a compromised AWS credential in a Terraform run (if a malicious user gains access to an AWS credential used in a Terraform run, that credential is only value for the length of the token's TTL),

  3. any changes to a role's permission could be managed through a Vault role rather than the distribution/management of static AWS credentials,

  4. enables development to provision resources without managing local, static AWS credentials

However, this approach may run into issues when applied to large multi-resource configurations. The generated dynamic AWS Credentials is only valid for the length of the token's TTL. As a result, if:

  1. the apply process exceeds than the TTL and the configuration needs to provision another resource or

  2. the apply confirmation time exceeds the TTL

the apply process will fail because the short-lived AWS Credentials have expired.

You could increase the TTL to conform to your situation; however, this also increases how long the temporary AWS credentials are, increasing the malicious actor's attack surface.

»Summary

Congratulations! You have successfully:

  1. configured Vault's AWS Secret Engine through Terraform,
  2. used dynamic short-lived AWS credentials to provision infrastructure, and
  3. restricted the AWS credential's permissions by adjusting the corresponding Vault role

Remember to clean up environment by destroying all resources in both Vault Admin and Terraform Operator workspaces.

$ terraform destroy --auto-approve && cd ../vault-admin-workspace && terraform destroy --auto-approve

Remember to stop your local Vault instance used in this guide by hitting Ctrl + C in the terminal window running Vault.

Now that you have inject secrets into Terraform using the Vault provider, you may like to: