Inspecting data in Consul storage
This tutorial focuses on inspecting Vault data stored in Consul. Also, refer to Vault Limits and Maximums for known upper limits on the size of certain fields and objects, and configurable limits on others.
Vault creates and persists critical operational data, such as configuration, leases, roles, secrets, and write ahead logs to its storage. Gathering key facts about these operational data can often be helpful when engaged in advanced troubleshooting.
Vault stores operational data as key/value pairs in a tree like hierarchy that you can visualize and think of similarly to data organization in an OS filesystem.
Certain techniques used for inspecting and counting the data (such as recursive listing) are impossible solely within the context of the Vault HTTP API. With that in mind, this tutorial uses examples which instead gather details directly from the underlying storage.
The examples provided here are directly applicable to the Consul storage backend, and include flavors for both the Consul HTTP API and consul
CLI.
- Notes and prerequisites
- New Vault server data example
- Secrets engine data
- Token and accessor data
- Lease data
- What about Enterprise Namespaces?
- Size of data on disk
- Summary
- Help and Reference
Notes and prerequisites
Please pay attention to the following list of important notes and prerequisites before you begin.
This is not meant to be a comprehensive tutorial covering every possible item which you can find in Vault data. It's an introduction to help you get started in inspecting your Vault data in Consul.
Please be careful with the commands shown in this tutorial; to list all the keys in a particular key/value path can be resource intensive on Consul, so do think about the impact your measuring can have. Be extra careful if using these techniques to inspect data a production Consul cluster.
To inspect Vault data in a Consul storage backend you can use the Consul HTTP API with tools like cURL and jq to fetch the information and process it. If it's available, you can also use a consul binary and
consul kv
commands instead of the Consul HTTP API. It's fine to use a Consul Community Edition binary for the command line interactions in this tutorial.This tutorial uses examples from the root namespace; for details about inspecting data within other namespaces created using the Enterprise Namespaces feature, please see the What about Enterprise Namespaces? section.
The Consul server used requires an ACL token and uses TLS so URLs use the https protocol. Set the following environment variables so that additional command flags are not required.
CONSUL_HTTP_ADDR
: This value is theaddress:port
part (i.e.127.0.0.1:8500
) of the full URL.CONSUL_HTTP_TOKEN
: This value is an ACL token that provides suitable read capabilities for the Consul server cluster's key/value storeHere is an example of exporting those environment variables in the terminal session that you will use for the example commands.
New Vault server data example
You will get to know Vault data by viewing an actual example with descriptions of each element.
When Vault is first initialized and unsealed the persisted data will resemble this example taken from the output of the tree
utility on data migrated from Consul to a filesystem.
Note
These data represent that from a Vault version 1.7.2 server; your results could vary based on your Vault server version.
A total of 73 key/value pairs are present in this example, representing all the data necessary for Vault to begin operations. A Vault server that is in production will have considerably more data and key/value pairs related to its specific auth methods, secrets engines, and system mounts.
Here is a brief explanation of each major branch and the elements within them from example.
- core: Items contained in core are critical and internal to Vault operations; these include data about internal auditing, authentication, keyring, mounts, the root key, the seal configuration, cluster information, HSM barrier unseal keys, seal wrapping, and more.
- index: This is local index data used by the Performance Standby feature.
- index-dr: This is index data for the Disaster Recovery mode of Vault Enterprise Replication.
- logical: Dynamic secret configuration and static secrets live here.
- sys: System data includes policy configuration along with tokens and their accessors.
- wal: Write ahead logs (WAL) are present in Vault Enterprise installations to support the Performance Standby feature and assist with enabling Enterprise Replication.
Those are the basics for now.
You can continue by inspecting secrets engine data.
Secrets engine data
A common question about Vault secret data during support and operations troubleshooting scenarios is How many secrets do I have in Vault for this secrets engine?
To begin down the path of answering this question, you can develop some understanding of the secrets engine data storage structure with further examples.
Vault stores secrets engine data in the configured storage at the default path /vault/logical/<UUID>/
where <UUID>
represents a unique identifier for the specific enabled secrets engine.
If your Vault servers use a configured storage path value other than vault, you need to keep this in mind and use your actual path in the examples.
When you initialize and unseal a new Vault, the Identity secrets engine gets enabled in the storage as shown in the example data:
After Vault is further configured, the logical
path holds more secrets engine data. For example, here is a tree view of example secrets engine data with detailed explanation of each element.
The earlier example shows paths for several secrets engines in the root namespace; here are details on each secrets engine and its associated elements:
- e2b7c3e2-3e21-3391-b73c-8a991a65789d A KV Secrets Engine - Version 2 containing internal configuration and metadata along with the secret data versions found under the
versions
key. - 2788376d-7042-4737-1ebd-9f6391a01f4e A PKI secrets engine which represents the root Certificate Authority (CA). It holds the CA information, The Certificate Revocation List (CRL) data, the URL configuration, internal configuration (with a CA bundle), and a role in this case called tacobot-root.
- b7183aba-6e64-e001-fe57-3e7e4508fc0c A PKI secrets engine which represents the intermediate CA. It holds the CA information, The CRL data, the URL configuration, internal configuration (with a CA bundle), and a role in this case called tacobot-int. Note also that it has a
certs
key with some certificate serial numbers present which represent the certificates issued from the tacobot-int role. - cb1bfb31-3ccb-ef29-6352-874902c3a021 A Database Secrets Engine with configuration and roles for MongoDB and MySQL.
- d1689597-4f78-a30b-7532-e7806be9fcba An Identity Secrets Engine is the identity management solution for Vault and enabled by default. You cannot disable or move this secrets engine.
- fbd73ad9-4f9c-45be-5be2-3758d04808af A Cubbyhole Secrets Engine gets enabled by default. You cannot disable, move, or enable Cubbyhole secrets engines.
Total secret count
You can get a total count of keys (including secrets and their containing keys) across all enabled secrets engines with a command like this:
The output that results should be just the count, for example:
The output shows that these Vault data contain about 40,000 static secrets (and their containing keys/folders).
Use consul kv
with a command like the following to get the same result:
Total count for all enabled secrets engines
You can get a nicely sorted count per enabled secrets engine with consul kv
like so:
This produces results like the following:
These results are somewhat atypical in their simplicity, but you could expect additional UUIDs and their key counts in a more densely configured Vault.
Total count for a single secrets engine
If you'd prefer a count from a specific secrets engine, you can alter the URL in your API query. For example, here is the command to grab the count of secrets in the Cubbyhole Secrets Engine:
Example output:
The Cubbyhole secrets engine with the UUID a54a02ec-e99c-bca7-248d-479a410159b8 has just the one secret.
You can also get a count per secrets engine with consul kv
like this:
The output matches the API call output as expected.
Token and accessor data
You can find active tokens and their accessors under vault/sys/token
.
This a an example tree of paths from a minimal Vault instance for purposes of illustration:
You can get the total count of all active tokens in Vault like this:
The output is the key count:
or with consul kv
:
Likewise, you can get count of token accessors this way:
The output is the key count:
or with consul kv
:
Lease data
The root path for Vault lease data in Consul is at vault/sys/expire/
.
Here is a an example tree of paths from a minimal Vault instance for purposes of illustration:
This represents all valid leases in Vault which will expire, organized by different auth methods and role paths. The earlier example shows leases for the root namespace including credentials from the Token Auth Method, the AppRole Auth Method, and MongoDB Database Secrets Engine.
You can get a total lease count with:
The output is the key count:
or with consul kv
:
Tip
This represents leases which will expire, not leases which have already expired. This count can be close to but not typically the same as the active tokens count.
If you want a count for a specific auth method, such as AppRole, use its path instead:
The output is the key count:
or with consul kv
:
Role path
If you're interested in the number of active leases associated with a given role, you can use the role's path.
For example, with the Database Secrets Engines, suppose that you have the following roles defined:
The roles will be present by name in the Consul key/value store at the following path:
You can count leases in this path with the Consul HTTP API like this:
The output is the key count:
or with consul kv
:
The output is the key count:
The output shows 8658 leases under the mongodb-readonly role at the moment.
What about Enterprise Namespaces?
Vault Enterprise version 0.11.0 introduced the Namespaces feature.
This changes the earlier procedures slightly, in that each namespace will encapsulate its own leases and tokens in paths under the namespace internal storage path name. Note that the internal name value is not the same as its user-configured name.
This an example tree of paths from a minimal Vault instance for purposes of illustration:
The actual user-configured namespace name is example-namespace
, and gets stored internally as a short unique identifier instead; for the earlier example, this value is 3E5Y0
.
Once you have determined the storage path for the namespace, you can then compose commands like the earlier ones against the root namespace with your namespaces.
For example, to get the lease count for all the AppRole leases in the example-namespace_/_3E5Y0
namespace with cURL, you must specify the user-specified namespace as the header X-Vault-Namespace
value:
The output is the key count:
or with consul kv
with the namespace storage path name:
The output is the key count:
Likewise for a count of active tokens:
The output is the key count:
or with consul kv
and the namespace storage path name:
The output is the key count:
Size of data on disk
Consul provides a handy way to get the size of all Vault secrets on disk in the consul kv export
command.
You can export the path corresponding to the keys (vault/logical/
by default) with a command like this:
Then you can inspect the size of the resulting JSON file for an idea of your secrets' size on disk:
The secrets total 1000, plus their containing key for the Key/Value secrets engine. You can use jq
and ls
to show the secret count and size on disk with this command:
which results in:
The output shows that my 1000 total secrets consume about 204KB on disk.
You can also use the techniques shown above, to narrow your consul kv export
to a particular secrets engine, thereby measuring just those secrets instead of all secrets in total.
You can also use this technique for tokens by exporting the proper token key space and measuring the resulting file size, too:
Retrieve the KV pairs at the vault/sys/token/id/
path and save the results in a file, tokens.json
.
The following command uses jq
to calculate the number of records in the tokens.json
.
Keep in mind that namespaces will alter the above paths. If you are unclear about this, please review the What about Enterprise Namespaces? section for more details.
Summary
In this article you learned about how Vault stores its operational data along with techniques for gathering basic information about them, like counts and sizes. You should be able to use what you have learned here as a starting point in inspecting and measuring the characteristics most important to you about your Vault data.