The Static Secrets tutorial introduced the basics of working with key-value secrets engine. This tutorial demonstrates the K/V Secrets Engine v2 with Secret Versioning.
»Challenge
The KV secrets engine v1 does not provide a way to version or roll back secrets. This made it difficult to recover from unintentional data loss or overwrite when more than one user is writing at the same path.
»Solution
Run the version 2 of KV secrets engine which can retain a configurable number of secret versions. This enables older versions' data to be retrievable in case of unwanted deletion or updates of the data. In addition, its Check-and-Set operations can be used to protect the data from being overwritten unintentionally.
»What this tutorial covers
This tutorial will walk you through the basic features of the KV v2 secrets engine:
- Step 1: Check the KV secrets engine version
- Step 2: Write secrets
- Step 3: Retrieve a specific version of secret
- Step 4: Specify the number of versions to keep
- Step 5: Delete versions of secret
- Step 6: Permanently delete data
- Step 7: Configure automatic data deletion
- Step 8: Check-and-Set operations
- Compare KV v1 and KV v2
»Prerequisites
To perform the tasks described in this tutorial, you need to have a Vault environment. Refer to the Getting Started tutorial to install Vault.
Launch Terminal
This tutorial includes a free interactive command-line lab that lets you follow along on actual cloud infrastructure.
»Policy requirements
NOTE: For the purpose of this tutorial, you can use root token to work
with Vault. However, it is recommended that root tokens are only used for
initial setup or in emergencies. As a best practice, use tokens with
appropriate set of policies based on your role in the organization.
To perform all tasks demonstrated in this tutorial, your policy must include the following permissions:
Vault version: The -output-policy flag requires Vault 1.11.0 or later.
To perform all tasks demonstrated in this tutorial, you are going to need to run
a number of vault CLI commands. Adding the -output-policy flag does not
execute the command, but instead prints the ACL policy needed to successfully
execute the command.
Check the policy needed for kv put:
Output:
Now, check the ACL policy needed for kv get:
If you are not familiar with policies, complete the policies tutorial.
»Lab setup
Open a terminal and start a Vault dev server with root as the root token.
The Vault dev server defaults to running at 127.0.0.1:8200. The server is
also initialized and unsealed.
Insecure operation: Do not run a Vault dev server in production. This approach is only used here to simplify the unsealing process for this demonstration.
Export an environment variable for the vault CLI to address the Vault server.
Export an environment variable for the vault CLI to authenticate with the
Vault server.
The Vault server is ready.
»Step 1: Check the KV secrets engine version
(Persona: admin)
The Vault server started in dev
mode, automatically enables v2
of the KV secrets engine at the secret/ path. Verify that KV secrets engine
is enabled and is set to version 2.
Display all the enabled secrets engine.
The results display a table of all enabled secrets engines. One entry in the
table has the Path of secret/ with the Type of kv. The Options
column displays the version number.
Enable KV v2 at secret/.
If the KV version is version:1, upgrade it to version:2.
»Step 2: Write secrets
To understand how the versioning works, let's write some test data.
Create a secret at the path secret/customer/acme with a name and
contact_email.
New in Vault 1.11.0: The following command is equivalent to the above command.
Vault 1.11.0 introduced an alternative command flag (-mount) to specify the
mount path of the secrets engine. See the Compare KV v1 and KV
v2 section for further discussion.
Example output:
The secret stores metadata along with the secret data. The version
is an auto-incrementing number that starts at 1.
Create secret at the same path secret/customer/acme but with different
secret data.
Example output:
The secret is fully replaced and the version is incremented to 2.
Get the secret defined at the path secret/customer/acme.
New in Vault 1.11.0: The following command is equivalent to the above command.
Example output:
The output displays version 2 of the secret. Creating a secret at the same path replaces the existing data; fields are not merged together. The secret data stored in earlier versions is still accessible but it is no longer returned by default.
»Patch the existing data
While vault kv put fully replaces the current version of the secret;
therefore, you need to send the entire set of data including the values that
remain the same. To partially update the current version of the secret, you can
use vault kv patch command instead.
Use case: Think of an application that does not have read permission, but
captures partial data updates. The vault kv patch command allows the
application to send only the changing values to Vault.
ACL Policy: KV v2 secrets engine honors the distinction between the
create and update capabilities inside ACL policies. The patch capability
is also supported which is used to represent partial updates whereas the
update capability represents full overwrites. To permit partial updates, be
sure to add patch capability in the ACL policy.
Update the contact_email value to admin@acme.com (current value is
john.smith@acme.com).
Example output:
The patch command creates a new version of the secret which merges the fields within the secret data.
Read the secret at secret/customer/acme to verify the change.
The contact_email value is updated to admin@acme.com, and the version is
now 3.
»Add custom metadata
You saw that the secret's key (in this example, secret/customer/acme) has
metadata associated with it. An organization may wish to add custom metadata
describing further details (technical contact, mission criticality, etc.) about
the secret.
When you are running Vault 1.9.0 or later, you can add custom metadata using
custom_metadata parameter. This feature stores a map of arbitrary
string-to-string valued user-provided metadata meant to describe the secret.
The -custom-metadata flag can be repeated to add multiple key-value pairs.
Output:
Now, when you read the secret, the returned secret metadata displays the custom metadata.
Example output:
»Step 3: Retrieve a specific version of secret
You may run into a situation where you need to view the secret before an update.
Get version 1 of the secret defined at the path secret/customer/acme.
Get the metadata of the secret defined at the path secret/customer/acme.
»Step 4: Specify the number of versions to keep
By default, the kv-v2 secrets engine keeps up to 10 versions. Let's limit the
maximum number of versions to keep to be 4.
Configure the secrets engine at path secret/ to limit all secrets to a
maximum of 4 versions.
Every secret stored for this engine are set to a maximum of 4 versions.
Display the secrets engine configuration settings.
Configure the secret at path secret/customer/acme to limit secrets to
a maximum of 4 versions.
The secret can also define the maximum number of versions.
Create four more secrets at the path secret/customer/acme.
Get the metadata of the secret defined at the path secret/customer/acme.
Example output:
The metadata displays the current_version and the history of versions stored.
Secrets stored at this path are limited to 4 versions. Version 1 and 2 are
deleted.
Verify that version 1 of the secret defined at the path secret/customer/acme
are deleted.
»Step 5: Delete versions of secret
Delete version 4 and 5 of the secrets at path secret/customer/acme.
Get the metadata of the secret defined at the path secret/customer/acme.
The metadata on versions 4 and 5 reports its deletion timestamp
(deletion_time); however, the destroyed parameter is set to false.
Undelete version 5 of the secrets at path secret/customer/acme.
»Step 6: Permanently delete data
Destroy version 4 of the secrets at path secret/customer/acme.
Get the metadata of the secret defined at the path secret/customer/acme.
The metadata displays that Version 4 is destroyed.
Delete all versions of the secret at the path secret/customer/acme.
»Step 7: Configure automatic data deletion
As of Vault 1.2, you can configure the length of time before a version gets
deleted. For example, if your organization requires data to be deleted after 10
days from its creation, you can configure the K/V v2 secrets engine to do so by
setting the delete_version_after parameter.
For demonstration, configure the secrets at path secret/test to delete
versions after 40 seconds.
Create a secret at the path secret/test.
Again, create a secret at the path secret/test.
Again, create a secret at the path secret/test.
NOTE: You can use upper-arrow key to recover the previously executed command.
Get the metadata of the secret defined at the path secret/test.
Example output:
The metadata displays a deletion_time set on each version. After 40 seconds,
the data gets deleted automatically. The data has not been destroyed.
Get version 1 of the secret defined at the path secret/test.
»Step 8: Check-and-Set operations
The v2 of KV secrets engine supports a Check-And-Set operation to prevent
unintentional secret overwrite. When you pass the cas flag to Vault, it first
checks if the key already exists.
Display the secrets engine configuration settings.
The cas_required setting is false. The KV secrets engine defaults to disable
the Check-And-Set operation.
Configure the secrets engine at path secret/ to enable Check-And-Set.
Configure the secret at path secret/partner to enable Check-And-Set.
Once check-and-set is enabled, every write operation requires the cas
parameter with the current version of the secret. Set cas to 0 when a secret
at that path does not already exist.
Create a new secret at the path secret/partner.
Overwrite the secret at the path secret/partner.
Example output:
»Compare KV v1 and KV v2
The secret path used by the vault kv command and the actual API endpoint looks
slightly different when you are working with KV v2 secrets engine. In this
section, you are going to compare KV v1 and KV v2 to understand the difference
between the two.
Enable KV v1 secrets engine at kv-v1/.
Enable KV v2 secrets engine at kv-21/.
You can use -output-curl-string command to print an equivalent cURL command
string for any vault command. Run the vault kv command with the
-output-curl-string flag against KV v1 and v2.
The endpoint is /kv-v1/test which matches the command.
Now, run the same command against kv-v2 to compare the output.
The endpoint is /kv-v2/data/test although the CLI command queried for
kv-v2/test. The Vault CLI is a convenient wrapper around the API. Based on the
target secrets engine, it sends requests to the appropriate API endpoint.
This behavior was confusing. To reduce this confusion, Vault 1.11.0
introduced the -mount flag to change the way you interact with the KV secrets
engine. You specify where the target secrets engine is enabled at
(<mount_path>) and the secret key (key) you wish to access.
The following command is equivalent to vault kv put kv-v2/credentials.
The output tells you that the data was successfully written at
kv-v2/data/credentials.
»API endpoint comparison
The table lists the vault kv subcommands and their respecting API endpoint.
This assumes that the KV secrets engine is enabled at secret/.
| Command | KV v1 endpoint | KV v2 endpoint |
|---|---|---|
vault kv get | secret/<key_path> | secret/data/<key_path> |
vault kv put | secret/<key_path> | secret/data/<key_path> |
vault kv list | secret/<key_path> | secret/data/<key_path> |
vault kv delete | secret/<key_path> | secret/data/<key_path> |
vault kv patch | N/A | secret/data/<key_path> |
vault kv rollback | N/A | secret/data/<key_path> |
vault kv undelete | N/A | secret/undelete/<key_path> |
vault kv destroy | N/A | secret/destroy/<key_path> |
vault kv metadata | N/A | secret/metadata/<key_path> |
»Next steps
This tutorial demonstrated the versioned KV secrets engine (kv-v2) feature. To
integrate the KV secrets engine into your existing application, you must
implement the Vault API to accomplish that. Alternatively, you can leverage
Vault Agent which significantly reduces the amount of code change introduced to
your application.
The Vault Agent Templates tutorial provides an end-to-end example.
If you are not familiar with Consul Template, it is recommended to go through the Direct Application Integration tutorial first.
