Developer

Direct Application Integration

A modern system requires access to a multitude of secrets: database credentials, API keys for external services, credentials for service-oriented architecture communication, etc. Vault steps in to provide a centralized secret management system. The next step is to decide how your applications acquire the secrets from Vault.

This guide introduces Consul Template and Envconsul to help you determine if these tools speed up the integration of your applications once secrets are securely managed by Vault.

» Consul Template

Despite its name, Consul Template does not require a Consul cluster to operate. It retrieves secrets from Vault and manages the acquisition and renewal lifecycle.

» Envconsul

Envconsul launches a subprocess which dynamically populates environment variables from secrets read from Vault. Your applications then read those environment variables. Despite its name, Envconsul does not require a Consul cluster to operate. It enables flexibility and portability for applications across systems.

» Challenge

If your application code or script contains some secrets (e.g. database credentials), it makes a good sense to manage the secrets using Vault. However, it means that your application will need to retrieve the secrets at runtime. Does that mean the application developers must make some code change?

Is there an easy way to retrieve the secrets from Vault and populate the application code or script with secrets as needed?

» Solution

Both Consul Template and Envconsul provide first-class support for Vault. Leveraging these tools can minimize the level of changes introduced to your applications. Depending on the current application design, you may not need to make minimal to no code change.

» Prerequisites

To perform the tasks described in this guide, you need:

» PostgreSQL

This guide uses the database secrets engine to demonstrate the use of Consul Template and Envconsul. Therefore you need a PostgreSQL server to connect to.

» Policy requirements

To perform all tasks demonstrated in this guide, your policy must include the following permissions:

# Enable database secrets engines at "database/" path
path "sys/mounts/database" {
  capabilities = [ "create", "read", "update", "delete", "list" ]
}

# Configure the database secret engine and create roles
path "database/*" {
  capabilities = [ "create", "read", "update", "delete", "list" ]
}

# Write ACL policies
path "sys/policy/*" {
  capabilities = [ "create", "read", "update", "delete", "list" ]
}

# Manage tokens for Consul Template & Envconsul to use
path "auth/token/create" {
  capabilities = [ "create", "read", "update", "delete", "list", "sudo" ]
}

If you are not familiar with policies, complete the policies guide.

» Steps

This guide demonstrates the use of Consul Template and Envconsul tools. To understand the difference between the two tools, you are going to retrieve the same information from Vault.

  1. Setup Database Secrets Engine
  2. Generate Client Token
  3. Use Consul Template to Populate DB Credentials
  4. Use Envconsul to Retrieve DB Credentials

» Step 1: Setup Database Secrets Engine

In this step, you are going to enable and configure the database secrets engine using postgresql-database-plugin where the database connection URL is postgresql://root:rootpassword@localhost:5432/myapp.

» CLI command

# First, enable the database secrets engine
$ vault secrets enable database

# Configure the secret engine with appropriate parameter values
$ vault write database/config/postgresql
      plugin_name=postgresql-database-plugin \
      allowed_roles=* \
      connection_url=postgresql://root:rootpassword@localhost:5432/myapp

# Create readonly.sql to define a role permission in SQL
$ tee readonly.sql <<EOF
CREATE ROLE "{{name}}" WITH LOGIN PASSWORD '{{password}}' VALID UNTIL '{{expiration}}';
GRANT SELECT ON ALL TABLES IN SCHEMA public TO "{{name}}";
EOF

# Create a role, "readonly"
$ vault write database/roles/readonly db_name=postgresql creation_statements=@readonly.sql \
    default_ttl=1h max_ttl=24h

» API call using cURL

# Enable `database` secret engine using `/sys/mounts` endpoint
$ curl --header "X-Vault-Token: ..." \
       --request POST \
       --data '{"type":"database"}' \
       https://127.0.0.1:8200/v1/sys/mounts/database

# Specify the database connection URL according to your environment
$ tee payload.json <<EOF
{
    "plugin_name": "postgresql-database-plugin",
    "allowed_roles": "*",
    "connection_url": "postgresql://root:rootpassword@localhost:5432/myapp"
}
EOF

# Configure the database secrets engine by passing the request payload
$ curl --header "X-Vault-Token: ..." \
       --request POST \
       --data @payload.json \
       http://127.0.0.1:8200/v1/database/config/postgresql

# Create the request payload to create a role
$ tee payload.json <<EOF
{
  "db_name": "postgres",
  "creation_statements": ["CREATE ROLE \"{{name}}\" WITH LOGIN PASSWORD '{{password}}' VALID UNTIL '{{expiration}}';
   GRANT SELECT ON ALL TABLES IN SCHEMA public TO \"{{name}}\";"],
  "default_ttl": "1h",
  "max_ttl": "24h"
}
EOF

# Create a role named readonly
$ curl --header "X-Vault-Token: ..." \
       --request POST \
       --data @payload.json \
       http://127.0.0.1:8200/v1/database/roles/readonly

» Step 2: Generate Client Token

Consul Template tool itself is a Vault client. Therefore it must have a valid token with policies permitting it to retrieve secrets from database secret engine you just configured in Step 1.

First, create a policy definition file, db_creds.hcl. This policy allows read operation on the database/creds/readonly path to obtain the dynamically generated username and password to access the PostgreSQL database. In addition, the policy allows renewal of the lease if necessary.

db_creds.hcl:

path "database/creds/readonly" {
  capabilities = [ "read" ]
}

path "/sys/leases/renew" {
  capabilities = [ "update" ]
}

Now, create a policy named db_creds and generate a token with this policy attached.

» CLI Command

# Create a `db_creds` policy
$ vault policy write db_creds db_creds.hcl

# Create a token with db_creds policy:
$ vault token create -policy="db_creds"
Key                  Value
---                  -----
token                89956bf1-6f4d-435d-4cf3-7496e9520a87
token_accessor       319eddff-42a1-eb2b-801e-dd8a0c0b07b4
token_duration       768h
token_renewable      true
token_policies       ["db_creds" "default"]
identity_policies    []
policies             ["db_creds" "default"]

» API call using cURL

# Create an API request payload
$ tee payload.json <<EOF
{
  "policy": "path \"database/creds/readonly\" {\n capabilities = [ \"read\" ]\n } \n path \"sys/leases/renew\" {\n capabilities = [ \"update\" ] \n}"
}
EOF

# Create db_creds policy
$ curl --header "X-Vault-Token: ..." \
       --request PUT \
       --data @payload.json \
       http://127.0.0.1:8200/v1/sys/policy/db_creds

# Generate a new token with db_creds policy
$ curl --header "X-Vault-Token: ..." \
       --request POST \
       --data '{"policies": ["db_creds"]}' \
       http://127.0.0.1:8200/v1/auth/token/create | jq
{
  ...
   "auth": {
     "client_token": "37413eca-96aa-1d47-d09d-d1cad322c419",
     "accessor": "a05aa3ce-b5d6-9e82-da7f-181d78d475e4",
     "policies": [
       "db_creds",
       "default"
     ],
     "token_policies": [
       "db_creds",
       "default"
     ],
     ...
}

» Step 3: Use Consul Template to Populate DB Credentials

Assume that your application requires PostgreSQL database credentials to read data. Its configuration file, config.yml looks like:

username: '<DB_USRENAME>'
password: '<DB_PASSWORD>'
database: 'myapp'

To have Consul Template to populate the <DB_USRENAME> and <DB_PASSWORD>, you need to create a template file with Consul Template templating language.

Create a template file by replacing the username and password with Consul Template syntax and save it as config.yml.tpl. The file should contain the following:

---
{{- with secret "database/creds/readonly" }}
username: "{{ .Data.username }}"
password: "{{ .Data.password }}"
database: "myapp"
{{- end }}

Execute the consul-template command to populate config.yml file.

The Consul Template command is: consul-template -template="<input_file>:<output_file>"

The input file is the config.yml.tpl and specify the desired output file name to be config.yml:

$ VAULT_TOKEN=<token> consul-template -template="config.yml.tpl:config.yml" -once

While <token> is the token you copied at Step 2.

Open the generated config.yml file to verify its content. It should look similar to:

$ cat config.yml
---
username: "v-token-readonly-tu17xrtz345uz643980r-1527630039"
password: "A1a-7s0z9y223x2rp6v9"
database: "myapp"

The username and password were retrieved from Vault and populated in the config.yml file.

» Step 4: Use Envconsul to Retrieve DB Credentials

Create a file named app.sh containing the following:

#!/usr/bin/env bash

cat <<EOT
My connection info is:

username: "${DATABASE_CREDS_READONLY_USERNAME}"
password: "${DATABASE_CREDS_READONLY_PASSWORD}"
database: "my-app"
EOT

The main difference here is that the app.sh is reading environment variables to set username and password values; therefore no templating is involved.

Run the Envconsul tool using the Vault token you generated at Step 2.

$ VAULT_TOKEN=<token> envconsul -upcase -secret database/creds/readonly ./app.sh

My connection info is:

username: "v-token-readonly-ww1tq33s7z5uprpxxy68-1527631219"
password: "A1a-u54wut0v605qwz95"
database: "my-app"

The output should display the username and password populated.

The -upcase flag tells Envconsul to convert all environment variable keys to uppercase. Otherwise, the default uses lowercase (e.g. database_creds_readonly_username).

» Help and Reference