Codify Management of Vault Enterprise Using Terraform

»Challenge

A manual system administration can become a challenge as the scale of infrastructure increases. Often, an organization must manage multiple Vault environments (development, testing, staging, production, etc.). Keeping up with the increasing management demand soon becomes a challenge without some sort of automation.

»Solution

One of the pillars behind the Tao of Hashicorp is automation through codification.

HashiCorp Terraform is an infrastructure as code which enables the operation team to codify the Vault configuration tasks such as the creation of policies. Automation through codification allows operators to increase their productivity, move quicker, promote repeatable processes, and reduce human error.

This tutorial demonstrates techniques for creating Vault policies and configurations using Terraform Vault Provider.

»Prerequisites

• Terraform installed

• Vault 1.4 or later

  To test and learn the use of Vault provider, you can use a Vault server running in development mode since you are going to perform Vault admin tasks in this tutorial.

Online tutorial: An interactive tutorial is also available if you do not wish to install a Vault HA cluster locally. Click the Show Terminal button to start.

»Scenario introduction

Vault administrators must manage multiple Vault environments. The test servers get destroyed at the end of each test cycle and a new set of servers must be provisioned for the next test cycle. To automate the Vault server configuration, you are going to use Terraform to provision the following Vault resources.

The admins policy must be created in all namespaces: root, finance, and engineering. The expected admin tasks are the same across the namespaces.

»Examine the Terraform files

Clone or download the demo assets from the hashicorp/vault-guides GitHub repository to perform the steps described in this tutorial.

To clone the repository, use the git clone command.

$ git clone https://github.com/hashicorp/vault-guides.git

$ git clone https://github.com/hashicorp/vault-guides.git

Alternatively, you can download the repository.

This repository contains supporting content for all of the Vault learn guides. The content specific to this tutorial can be found within a sub-directory.

Change the working directory to /operations/codify-mgmt/enterprise.

$ cd operations/codify-mgmt/enterprise

$ cd operations/codify-mgmt/enterprise

The directory contains Terraform files to configure Vault.

$ tree
.
├── auth.tf
├── main.tf
├── policies
│   ├── admin-policy.hcl
│   └── fpe-client-policy.hcl
├── policies.tf
└── secrets.tf

$ tree.├── auth.tf├── main.tf├── policies│   ├── admin-policy.hcl│   └── fpe-client-policy.hcl├── policies.tf└── secrets.tf

»Review main.tf

Open the main.tf file in your preferred text editor to examine its content. It defines two provider blocks each pointing to a different namespace.

provider "vault" {
  alias = "finance"
  namespace = "finance"
}

provider "vault" {
  alias = "engineering"
  namespace = "engineering"
}

provider "vault" {  alias = "finance"  namespace = "finance"}
provider "vault" {  alias = "engineering"  namespace = "engineering"}

The following blocks create finance and engineering namespaces.

resource "vault_namespace" "finance" {
  path = "finance"
}

resource "vault_namespace" "engineering" {
  path = "engineering"
}

resource "vault_namespace" "finance" {  path = "finance"}
resource "vault_namespace" "engineering" {  path = "engineering"}

If you are not familiar with Vault Enterprise namespace, refer to the Secure Multi-Tenancy with Namespaces tutorial.

»Nested namespaces

From line 54, it defines nested namespaces as shown below.

├── education
│   └── training
│       ├── boundary
│       └── vault_cloud
├── engineering
└── finance

├── education│   └── training│       ├── boundary│       └── vault_cloud├── engineering└── finance

First, create the top-level namespace, education namespace. To create a child namespace under education, specify provider which points to education namespace provider. Similarly, specify provider which points to training to create child namespaces, education/training/vault_cloud and education/training/boundary.

resource "vault_namespace" "education" {
  path = "education"
}

provider "vault" {
  alias = "education"
  namespace = trimsuffix(vault_namespace.education.id, "/")
}

# Create a childnamespace, 'training' under 'education'
resource "vault_namespace" "training" {
  provider = vault.education
  path = "training"
}

provider "vault" {
  alias = "training"
  namespace = trimsuffix(vault_namespace.training.id, "/")
}

# Create a childnamespace, 'vault_cloud' and 'boundary' under 'education/training'
resource "vault_namespace" "vault_cloud" {
  provider = vault.training
  path = "vault_cloud"
}

provider "vault" {
  alias = "vault_cloud"
  namespace = trimsuffix(vault_namespace.vault_cloud.id, "/")
}

# Create 'education/training/boundary' namespace
resource "vault_namespace" "boundary" {
  provider = vault.training
  path = "boundary"
}

provider "vault" {
  alias = "boundary"
  namespace = trimsuffix(vault_namespace.boundary.id, "/")
}

resource "vault_namespace" "education" {  path = "education"}
provider "vault" {  alias = "education"  namespace = trimsuffix(vault_namespace.education.id, "/")}
# Create a childnamespace, 'training' under 'education'resource "vault_namespace" "training" {  provider = vault.education  path = "training"}
provider "vault" {  alias = "training"  namespace = trimsuffix(vault_namespace.training.id, "/")}
# Create a childnamespace, 'vault_cloud' and 'boundary' under 'education/training'resource "vault_namespace" "vault_cloud" {  provider = vault.training  path = "vault_cloud"}
provider "vault" {  alias = "vault_cloud"  namespace = trimsuffix(vault_namespace.vault_cloud.id, "/")}
# Create 'education/training/boundary' namespaceresource "vault_namespace" "boundary" {  provider = vault.training  path = "boundary"}
provider "vault" {  alias = "boundary"  namespace = trimsuffix(vault_namespace.boundary.id, "/")}

This allows you to leverage multiple namespaces during the Vault configuration.

»Review policies.tf

Open the policies.tf file and examine the vault_policy resources.

# Create fpe-client policy in the root namespace
resource "vault_policy" "fpe_client_policy" {
  name   = "fpe-client"
  policy = file("policies/fpe-client-policy.hcl")
}

# Create admin policy in the root namespace
resource "vault_policy" "admin_policy" {
  name   = "admins"
  policy = file("policies/admin-policy.hcl")
}

# Create admin policy in the finance namespace
resource "vault_policy" "admin_policy_finance" {
  provider = vault.finance
  depends_on = [vault_namespace.finance]
  name   = "admins"
  policy = file("policies/admin-policy.hcl")
}

# Create admin policy in the engineering namespace
resource "vault_policy" "admin_policy_engineering" {
  provider = vault.engineering
  depends_on = [vault_namespace.engineering]
  name   = "admins"
  policy = file("policies/admin-policy.hcl")
}

# Create admin policy in the education namespace
resource "vault_policy" "admin_policy_education" {
  provider = vault.education
  depends_on = [vault_namespace.education]
  name   = "admins"
  policy = file("policies/admin-policy.hcl")
}

# Create admin policy in the 'education/training' namespace
resource "vault_policy" "admin_policy_training" {
  provider = vault.training
  depends_on = [vault_namespace.training]
  name   = "admins"
  policy = file("policies/admin-policy.hcl")
}

# Create admin policy in the 'education/training/vault_cloud' namespace
resource "vault_policy" "admin_policy_vault_cloud" {
  provider = vault.vault_cloud
  depends_on = [vault_namespace.vault_cloud]
  name   = "admins"
  policy = file("policies/admin-policy.hcl")
}

# Create admin policy in the 'education/training/boundary' namespace
resource "vault_policy" "admin_policy_boundary" {
  provider = vault.boundary
  depends_on = [vault_namespace.boundary]
  name   = "admins"
  policy = file("policies/admin-policy.hcl")
}

# Create fpe-client policy in the root namespaceresource "vault_policy" "fpe_client_policy" {  name   = "fpe-client"  policy = file("policies/fpe-client-policy.hcl")}
# Create admin policy in the root namespaceresource "vault_policy" "admin_policy" {  name   = "admins"  policy = file("policies/admin-policy.hcl")}
# Create admin policy in the finance namespaceresource "vault_policy" "admin_policy_finance" {  provider = vault.finance  depends_on = [vault_namespace.finance]  name   = "admins"  policy = file("policies/admin-policy.hcl")}
# Create admin policy in the engineering namespaceresource "vault_policy" "admin_policy_engineering" {  provider = vault.engineering  depends_on = [vault_namespace.engineering]  name   = "admins"  policy = file("policies/admin-policy.hcl")}
# Create admin policy in the education namespaceresource "vault_policy" "admin_policy_education" {  provider = vault.education  depends_on = [vault_namespace.education]  name   = "admins"  policy = file("policies/admin-policy.hcl")}
# Create admin policy in the 'education/training' namespaceresource "vault_policy" "admin_policy_training" {  provider = vault.training  depends_on = [vault_namespace.training]  name   = "admins"  policy = file("policies/admin-policy.hcl")}
# Create admin policy in the 'education/training/vault_cloud' namespaceresource "vault_policy" "admin_policy_vault_cloud" {  provider = vault.vault_cloud  depends_on = [vault_namespace.vault_cloud]  name   = "admins"  policy = file("policies/admin-policy.hcl")}
# Create admin policy in the 'education/training/boundary' namespaceresource "vault_policy" "admin_policy_boundary" {  provider = vault.boundary  depends_on = [vault_namespace.boundary]  name   = "admins"  policy = file("policies/admin-policy.hcl")}

»Review auth.tf

Open the auth.tf file and note that it enables userpass auth method and create a user, "student" with admins and fpe-client policies attached. The password is set to "changeme".

#--------------------------------
# Enable userpass auth method
#--------------------------------
resource "vault_auth_backend" "userpass" {
  type = "userpass"
}

resource "vault_generic_endpoint" "student" {
  depends_on           = [vault_auth_backend.userpass]
  path                 = "auth/userpass/users/student"
  ignore_absent_fields = true

  data_json = <<EOT
{
  "policies": ["fpe-client", "admins"],
  "password": "changeme"
}
EOT
}

#--------------------------------# Enable userpass auth method#--------------------------------resource "vault_auth_backend" "userpass" {  type = "userpass"}
resource "vault_generic_endpoint" "student" {  depends_on           = [vault_auth_backend.userpass]  path                 = "auth/userpass/users/student"  ignore_absent_fields = true
  data_json = <<EOT{  "policies": ["fpe-client", "admins"],  "password": "changeme"}EOT}

»Review secrets.tf

Open the secrets.tf file to note how it enables kv-v2 and transform secrets engines.

# Enable kv-v2 secrets engine in the finance namespace
resource "vault_mount" "kv-v2" {
  depends_on = [vault_namespace.finance]
  provider = vault.finance
  path = "kv-v2"
  type = "kv-v2"
}

# Transform secrets engine at root
resource "vault_mount" "mount_transform" {
  path = "transform"
  type = "transform"
}

# Enable kv-v2 secrets engine in the finance namespaceresource "vault_mount" "kv-v2" {  depends_on = [vault_namespace.finance]  provider = vault.finance  path = "kv-v2"  type = "kv-v2"}
# Transform secrets engine at rootresource "vault_mount" "mount_transform" {  path = "transform"  type = "transform"}

The following blocks create a new alphabet, template, transformation, and a role.

# Create an alphabet
resource "vault_transform_alphabet" "numerics" {
  path = vault_mount.mount_transform.path
  name = "numerics"
  alphabet = "0123456789"
}

# Create a transformation template
resource "vault_transform_template" "ccn" {
  path = vault_mount.mount_transform.path
  name = "ccn"
  type = "regex"
  pattern = "(\\d{4})-(\\d{4})-(\\d{4})-(\\d{4})"
  alphabet = vault_transform_alphabet.numerics.name
}

# Create a transformation named ccn-fpe
resource "vault_transform_transformation" "ccn-fpe" {
  path = vault_mount.mount_transform.path
  name = "ccn-fpe"
  type = "fpe"
  template = vault_transform_template.ccn.name
  tweak_source = "internal"
  allowed_roles = ["payments"]
}

# Create a role named 'payments'
resource "vault_transform_role" "payments" {
  path = vault_mount.mount_transform.path
  name = "payments"
  transformations = [vault_transform_transformation.ccn-fpe.name]
}

# Create an alphabetresource "vault_transform_alphabet" "numerics" {  path = vault_mount.mount_transform.path  name = "numerics"  alphabet = "0123456789"}
# Create a transformation templateresource "vault_transform_template" "ccn" {  path = vault_mount.mount_transform.path  name = "ccn"  type = "regex"  pattern = "(\\d{4})-(\\d{4})-(\\d{4})-(\\d{4})"  alphabet = vault_transform_alphabet.numerics.name}
# Create a transformation named ccn-fperesource "vault_transform_transformation" "ccn-fpe" {  path = vault_mount.mount_transform.path  name = "ccn-fpe"  type = "fpe"  template = vault_transform_template.ccn.name  tweak_source = "internal"  allowed_roles = ["payments"]}
# Create a role named 'payments'resource "vault_transform_role" "payments" {  path = vault_mount.mount_transform.path  name = "payments"  transformations = [vault_transform_transformation.ccn-fpe.name]}

Finally, lines 60 through 68 test the FPE transformation configured.

data "vault_transform_encode" "encoded" {
  path = vault_transform_role.payments.path
  role_name = "payments"
  value = "1111-2222-3333-4444"

  depends_on = [vault_transform_role.payments]
}

output "encoded" {
  value = data.vault_transform_encode.encoded.encoded_value
}

data "vault_transform_encode" "encoded" {  path = vault_transform_role.payments.path  role_name = "payments"  value = "1111-2222-3333-4444"
  depends_on = [vault_transform_role.payments]}
output "encoded" {  value = data.vault_transform_encode.encoded.encoded_value}

»Run Terraform to configure Vault

1. Set the client token in the VAULT_TOKEN environment variable. If not using the root token, be sure that this token has permissions to create policies, enable secrets engines, and enable auth methods.

   $ export VAULT_TOKEN="<token>"
   

   $ export VAULT_TOKEN="<token>"

   Copy

2. Set the target Vault server address in the VAULT_ADDR environment variable if it's not done so already.

   $ export VAULT_ADDR="http://127.0.0.1:8200"
   

   $ export VAULT_ADDR="http://127.0.0.1:8200"

   Copy

3. Initialize Terraform to pull Vault provider plugin.

   $ terraform init
   

   $ terraform init

   Copy

   This downloads the Vault plugin. When it completes, it displays a message, Terraform has been successfully initialized!

4. Execute the apply command to configure Vault.

   $ terraform apply
   

   $ terraform apply

   Copy

   This displays the actions to be performed by Terraform.

   When prompted, enter yes to accept the plan and proceed with Vault configuration.

   Plan: 22 to add, 0 to change, 0 to destroy.
   
   Do you want to perform these actions?
     Terraform will perform the actions described above.
     Only 'yes' will be accepted to approve.
   
     Enter a value: yes
   

   Plan: 22 to add, 0 to change, 0 to destroy.
   Do you want to perform these actions?  Terraform will perform the actions described above.  Only 'yes' will be accepted to approve.
     Enter a value: yes

   Once completed, the output similar to the following displays.

   Apply complete! Resources: 22 added, 0 changed, 0 destroyed.
   
   Outputs:
   
   encoded = 9704-7756-1680-5598
   

   Apply complete! Resources: 22 added, 0 changed, 0 destroyed.
   Outputs:
   encoded = 9704-7756-1680-5598

»Verify the configuration

»Clean up

When you are done exploring, you can undo the configuration made by Terraform.

First log back in with the client token used to run the terraform commands.

$ vault login $VAULT_TOKEN

$ vault login $VAULT_TOKEN

Destroy the Vault resources created by Terraform.

$ terraform destroy -auto-approve

...snip...

Destroy complete! Resources: 14 destroyed.

$ terraform destroy -auto-approve
...snip...
Destroy complete! Resources: 14 destroyed.

Remove the terraform state files.

$ rm *.tfstate.*

$ rm *.tfstate.*

»Next steps

Treat your Terraform files like any other code and manage them through a version control system such as GitHub. You may integrate it with your favorite CI/CD tool (Jenkins, Travis CI, Circle CI, etc.), always review and test the configuration.

Travis CI example:

You can test your Terraform files against a development server that runs locally, or use a Docker image of Vault.

sudo: false

env:
  - VAULT_ADDR=<target_vault_address>  VAULT_TOKEN=<token>

before_install:
  - scripts/install.sh

before_script:
  - vault server -dev -dev-root-token-id="$VAULT_TOKEN"

script:
  - terraform init
  - terraform apply -auto-approve

sudo: false
env:  - VAULT_ADDR=<target_vault_address>  VAULT_TOKEN=<token>
before_install:  - scripts/install.sh
before_script:  - vault server -dev -dev-root-token-id="$VAULT_TOKEN"
script:  - terraform init  - terraform apply -auto-approve

»Help and Reference

• Terraform Vault Provider documentation page
• Terraform Provider GitHub repository
• Learn Terraform