Static Secrets: Key/Value Secrets Engine

Vault can be used to store any secret in a secure manner. The secrets may be SSL certificates and keys for your organization's domain, credentials to connect to a corporate database server, etc. Storing such sensitive information in plaintext is not desirable.

NOTE: This tutorial demonstrates the use of Key/Value Secrets Engine Version 1. To learn about Key/Value Secrets Engine Version 2, refer to the Versioned Key/Value Secrets Engine tutorial.

»Personas

The end-to-end scenario described in this tutorial involves two personas:

devops with privileged permissions to write secrets
apps reads the secrets from Vault

»Challenge

Consider the following situations:

Developers use a single admin account to access a third-party app (e.g. Splunk) and anyone who knows the user ID and password can log in as an admin
SSH keys to connect to remote machines are shared and stored as a plaintext
API keys to invoke external system APIs are stored as a plaintext
An app integrates with LDAP, and its configuration information is in a plaintext

Organizations often seek an uniform workflow to securely store this sensitive information.

»Solution

Use Vault as centralized secret storage to secure any sensitive information. Vault encrypts these secrets using 256-bit AES in GCM mode with a randomly generated nonce prior to writing them to its persistent storage. The storage backend never sees the unencrypted value, so even if an attacker gained access to the raw storage, they wouldn't be able to read your secrets.

»Prerequisites

To perform the tasks described in this tutorial, you need to have a Vault environment. Refer to the Getting Started tutorial to install Vault. Make sure that your Vault server has been initialized and unsealed.

NOTE: An interactive tutorial is also available if you do not have a Vault environment to perform the steps described in this tutorial. Click the Show Terminal button to start.

»Policy requirements

NOTE: For the purpose of this tutorial, you can use the root token to work with Vault. However, it is recommended that root tokens are only used for just enough initial setup or in emergencies. As a best practice, use tokens with appropriate set of policies based on your role in the organization.

To perform all tasks demonstrated in this tutorial, your policy must include the following permissions:

# Enable key/value secrets engine at the kv-v1 path
path "sys/mounts/kv-v1" {
  capabilities = [ "update" ]
}

# To list the available secrets engines
path "sys/mounts" {
  capabilities = [ "read" ]
}

# Write and manage secrets in key/value secrets engine
path "kv-v1/*" {
  capabilities = [ "create", "read", "update", "delete", "list" ]
}

# Create policies to permit apps to read secrets
path "sys/policies/acl/*" {
  capabilities = [ "create", "read", "update", "delete", "list" ]
}

# Create tokens for verification & test
path "auth/token/create" {
  capabilities = [ "create", "update", "sudo" ]
}

If you are not familiar with policies, complete the policies tutorial.

»Scenario Introduction

This tutorial demonstrates the basic steps to store secrets using Vault. The scenario here is to store the following secrets:

API key (Google API)
Root certificate of a production database (MySQL)

To store your API key within the configured physical storage for Vault, use the key/value secrets engine.

The Key/Value secrets engine passes any operation through to the configured storage backend for Vault. For example, if your Vault server is configured with Consul as its storage backend, a "read" operation turns into a read from Consul at the same path.

You will perform the following:

Enable KV Secrets Engine
Store the Google API key
Store the root certificate for MySQL
Generate a token for apps
Retrieve the secrets

Personas Introduction

»Step 1: Enable KV Secrets Engine

Currently, when you start the Vault server in dev mode, it automatically enables v2 of the KV secrets engine at secret/.

This tutorial focuses on key/value v1 secrets engine. The Versioned Key/Value Secret Engin tutorial highlights features that are specific to the key/value v2 secrets engine.

Execute the following command to list enabled secrets engines.

$ vault secrets list -detailed

Under the Options, the version number is indicated.

Path          Type         Accessor           ...   Options           Description
----          ----         --------                 -------           -----------
cubbyhole/    cubbyhole    cubbyhole_9d52aeac ...   map[]             per-token private secret storage
identity/     identity     identity_acea5ba9  ...   map[]             identity store
secret/       kv           kv_2226b7d3        ...   map[version:2]    key/value secret storage
...

Execute the following command to enable the key/value secrets engine v1 at kv-v1/.

$ vault secrets enable -path="kv-v1" kv

»Step 2: Store the Google API key

Everything after the kv-v1 path is a key-value pair to write to the secrets engine. You can specify multiple values. If the value has a space, you need to surround it with quotes. Having keys with spaces is permitted, but strongly discouraged because it can lead to unexpected client-side behavior.

Let's assume that the path convention in your organization is kv-v1/<OWNER>/apikey/<APP> for API keys. To store the Google API key used by the engineering team, the path would be kv-v1/eng/apikey/Google. If you have an API key for New Relic owned by the DevOps team, the path would look like kv-v1/devops/apikey/New_Relic.

To set key/value secrets:

$ vault kv put kv-v1/<PATH> <KEY>=VALUE>

The <PATH> can be anything you want it to be, and your organization should decide on the naming convention that makes most sense.

Example:

$ vault kv put kv-v1/eng/apikey/Google key=AAaaBBccDDeeOTXzSMT1234BB_Z8JzG7JkSVxI
Success! Data written to: kv-v1/eng/apikey/Google

The secret key is "key" and its value is "AAaaBBccDDeeOTXzSMT1234BB_Z8JzG7JkSVxI" in this example.

Execute the following command to read back the secrets.

$ vault kv get kv-v1/eng/apikey/Google
=== Data ===
Key    Value
---    -----
key    AAaaBBccDDeeOTXzSMT1234BB_Z8JzG7JkSVxI

»Step 3: Store the root certificate for MySQL

For the purpose of this tutorial, generate a mock certificate using OpenSSL.

$ openssl req -x509 -sha256 -nodes -newkey rsa:2048 -keyout selfsigned.key -out cert.pem

The generated cert.pem file will look something like this.

NOTE: If you don't have OpenSSL, copy the above certificate and save it as cert.pem.

The command is basically the same as the Google API key example. The path convention for certificates is kv-v1/<ENVIRONMENT>/cert/<SYSTEM>. To store the root certificate for production MySQL, the path becomes kv-v1/prod/cert/mysql.

$ vault kv put kv-v1/prod/cert/mysql cert=@cert.pem

This example reads the root certificate from a PEM file from the disk, and store it under the kv-v1/prod/cert/mysql path.

NOTE: Any value that begins with @ indicates a file name. Data will be loaded from this file.

»Step 4: Generate a token for apps

To read the secrets, apps persona needs "read" ability on those secrets engine paths. In this scenario, the apps policy must include the following.

Example: apps-policy.hcl

# Read-only permit
path "kv-v1/eng/apikey/Google" {
  capabilities = [ "read" ]
}

# Read-only permit
path "kv-v1/prod/cert/mysql" {
  capabilities = [ "read" ]
}

First create apps policy

$ vault policy write apps apps-policy.hcl
Policy 'apps' written.

Create a new token with apps policy attached.

$ vault token create -policy="apps"
Key             Value
---             -----
token           e4bdf7dc-cbbf-1bb1-c06c-6a4f9a826cf2
token_accessor  54700b7e--data828-a6c4-6141-96e71e002bd7
token_duration  768h0m0s
token_renewable true
token_policies  [apps default]

Now apps can use this token to read the secrets.

NOTE: For the purposes of this tutorial, you created a policy for the apps persona, and generated a token for it. However, in the real world, you may have a dedicated policy author, or admin to write policies. Also, the consumer of the API key may be different from the consumer of the root certificate. Then each persona would have a policy based on what it needs to access.

Personas Introduction

»Step 5: Retrieve the secrets

Using the token from Step 4, read the Google API key and the root certificate for MySQL.

The command to read a secret is:

$ vault kv get kv-v1/<PATH>

Example:

Authenticate with Vault using the generated token first.

$ vault login e4bdf7dc-cbbf-1bb1-c06c-6a4f9a826cf2

Read the secret.

$ vault kv get kv-v1/eng/apikey/Google
Key                 Value
---                 -----
refresh_interval    768h0m0s
key                 AAaaBBccDDeeOTXzSMT1234BB_Z8JzG7JkSVxI

To return the key's value and nothing else, pass -field=key as an argument.

$ vault kv get -field=key kv-v1/eng/apikey/Google
AAaaBBccDDeeOTXzSMT1234BB_Z8JzG7JkSVxI

»Root certificate example:

The command is basically the same.

$ vault kv get -field=cert kv-v1/prod/cert/mysql
-----BEGIN RSA PRIVATE KEY-----
MIIEowIBAAKCAQEA6E2Uq0XqreZISgVMUu9pnoMsq+OoK1PI54rsA9vtDE6wiRk0GWhf5vD4DGf1
...

»Additional Discussion

»Q: How do I enter my secrets without exposing the secret in my shell's history?

As a precaution, you may wish to avoid passing your secret as a part of the CLI command so that the secret won't appear in the history file. Here are a few techniques you can use.

»Option 1: Use a dash "-"

An easy technique is to use a dash "-" and then press Enter. This allows you to enter the secret on a new line. After entering the secret, press Ctrl+d to end the pipe which will write the secret to the Vault.

$ vault kv put kv-v1/eng/apikey/Google key=-

AAaaBBccDDeeOTXzSMT1234BB_Z8JzG7JkSVxI
<Ctrl+d>

»Option 2: Read the secret from a file

Using the Google API key example, you can create a file containing the key (apikey.txt):

{
  "key": "AAaaBBccDDeeOTXzSMT1234BB_Z8JzG7JkSVxI"
}

The CLI command would look like:

$ vault kv put kv-v1/eng/apikey/Google @apikey.txt

»Option 3: Disable all vault command history

Sometimes, you may not even want the vault command itself to appear in history at all. Option 1 and Option 2 prevent the secret from appearing in your shell history. The vault command itself (vault kv put kv-v1/eng/apikey/Google) will still appear in your history.

You can configure your shell to avoid logging any vault commands to your history.

In bash:

$ export HISTIGNORE="&:vault*"

NOTE: This prevents vault commands from appearing when using the Up arrow or when searching command history with <Ctrl-r>.

»Q: How do I save multiple values at once?

The two examples introduced in this tutorial only had a single key/value pair. You can pass multiple values in a single command.

$ vault kv put kv-v1/dev/config/mongodb url=foo.example.com:35533 \
        db_name=users \
        username=admin password=passw0rd

Or, read several secrets from a file. First, create the JSON file containing the data.

$ tee mongodb.json <<EOF
{
    "url": "foo.example.com:35533",
    "db_name": "users",
    "username": "admin",
    "password": "pa$$w0rd"
}
EOF

Write secrets which will be read from the mongodb.json file.

$ vault kv put kv-v1/dev/config/mongodb @mongodb.json

»Help and Reference

Key/Value Secrets Engine
Key/Value Secrets Engine API
Client libraries for Vault API for commonly used languages

Infrastructure

Security

Networking

Applications