This tutorial focuses on inspecting Vault data stored in Consul. Also, refer to Vault Limits and Maximums for known upper limits on the size of certain fields and objects, and configurable limits on others.
Vault creates and persists critical operational data, such as configuration, leases, roles, secrets, and write ahead logs to its storage. Gathering key facts about these operational data can often be extremely helpful when engaged in advanced troubleshooting.
Vault stores operational data as key/value pairs in a tree like hierarchy that can be visualized and thought of in a similar manner as an OS filesystem with respect to data organization.
Certain techniques used for inspecting and counting the data (such as recursive listing) are not currently possible solely within the context of the Vault HTTP API. With that in mind, this tutorial uses examples that gather details directly from the the underlying storage instead.
The examples provided here are directly applicable to the Consul storage backend, and include flavors for both the Consul HTTP API and consul CLI.
- Notes and prerequisites
- New Vault server data example
- Secrets engine data
- Token and accessor data
- Lease data
- What about Enterprise Namespaces?
- Size of data on disk
- Summary
- Help and Reference
»Notes and prerequisites
Please pay attention to the following list of important notes and prerequisites before you begin.
This is not meant to be a comprehensive tutorial covering every possible type of data which could be found in Vault; rather, it is a means to help you get started in inspecting your Vault data in Consul.
Please be careful with the commands shown in this tutorial; to list all of the keys in a particular key/value path can be resource intensive on Consul, so do think about the impact your measuring can have, especially if using these techniques to inspect data a production Consul cluster.
To inspect Vault data in a Consul storage backend you can use the Consul HTTP API with tools like cURL and jq to fetch the information and process it. If it is available, you can also use a consul binary and
consul kvcommands instead of the Consul HTTP API. It is fine to use an open sourceconsulbinary for the command line interactions in this tutorial.This tutorial uses examples from the root namespace; for details about inspecting data within other namespaces created using the Enterprise Namespaces feature, please see the What about Enterprise Namespaces? section.
The Consul server used requires an ACL token and uses TLS so URLs use the https protocol. The following environment variables are set as a convenience so that additional command flags are not required.
CONSUL_HTTP_ADDR: This value is theaddress:portportion (i.e.127.0.0.1:8500) of the full URL only.CONSUL_HTTP_TOKEN: This value is an ACL token that provides suitable read capabilities for the Consul server cluster's key/value storeHere is an example of exporting those environment variables in the terminal session that we will use for the example commands.
»New Vault server data example
Let's get to know some Vault data by viewing an actual example with descriptions of each element.
When Vault is first initialized and unsealed the persisted data will resemble this example taken from the output of the tree utility on data which were migrated from Consul to a filesystem.
NOTE: These data represent that from a Vault version 1.7.2 server; your results could vary based on your Vault server version.
A total of 73 key/value pairs are present in this example, representing all of the data necessary for Vault to begin operations. A Vault server that is in production will have considerably more data and key/value pairs related to its specific auth methods, secrets engines, and so on.
Here is a brief explanation of each major branch and the elements within them from example.
- core: Items contained in core are critical and internal to Vault operations; these include data about internal auditing, authentication, keyring, mounts, the root key, the seal configuration, cluster information, HSM barrier unseal keys, seal wrapping, and more.
- index: This is local index data which can be used by the Performance Standby feature.
- index-dr: This is index data for the Disaster Recovery mode of Vault Enterprise Replication.
- logical: Dynamic secret configuration and static secrets are found here.
- sys: System data includes policy configuration along with tokens and their accessors.
- wal: Write ahead logs (WAL) are present in Vault Enterprise installations to support the Performance Standby feature and assist with enabling Enterprise Replication.
Those are the basics for now.
We will continue by inspecting secrets engine data.
»Secrets engine data
A common question about Vault secret data during support and operations troubleshooting scenarios is How many secrets do we currently have in Vault for this secrets engine?
To begin down the path of answering this question, let's develop some understanding of the secrets engine data storage structure with further examples.
Vault stores secrets engine data in the configured storage at the default path /vault/logical/<UUID>/ where <UUID> represents a unique identifier for the specific enabled secret backend.
If your Vault servers use a configured storage path value other than vault, you need to keep this in mind and use your actual path in the examples.
When a new Vault is initialized and unsealed, only the Identity secrets engine is configured and present in the storage as shown in the example data:
After Vault is further configured, the logical path would be expected to contain more secrets engine data. For example, here is a tree view of example secrets engine data with detailed explanation of each element.
The previous example shows paths for several secrets engines in the root namespace; here are details on each secrets engine and its associated elements:
- e2b7c3e2-3e21-3391-b73c-8a991a65789d A KV Secrets Engine - Version 2 containing internal configuration and metadata along with the secret data versions found under the
versionskey. - 2788376d-7042-4737-1ebd-9f6391a01f4e A PKI secrets engine which represents the root Certificate Authority (CA). It contains the CA information, The Certificate Revocation List (CRL) data, the URL configuration, internal configuration (with a CA bundle), and a role in this case called tacobot-root.
- b7183aba-6e64-e001-fe57-3e7e4508fc0c A PKI secrets engine which represents the intermediate CA. It contains the CA information, The CRL data, the URL configuration, internal configuration (with a CA bundle), and a role in this case called tacobot-int. Note also that it has a
certskey with some certificate serial numbers present which represent the certificates issued from the tacobot-int role. - cb1bfb31-3ccb-ef29-6352-874902c3a021 A Database Secrets Engine with configuration and roles for MongoDB and MySQL.
- d1689597-4f78-a30b-7532-e7806be9fcba An Identity Secrets Engine is the identity management solution for Vault and enabled by default. This secrets engine cannot be disabled or moved.
- fbd73ad9-4f9c-45be-5be2-3758d04808af A Cubbyhole Secrets Engine which is enabled by default. It cannot be disabled, moved, or enabled multiple times.
»Total secret count
You can get an approximate total count of keys (this includes secrets and their containing keys) across all currently enabled secrets engines with a command like this:
The output that results should be just the count, for example:
The output shows that these Vault data contain approximately 40,000 static secrets (and their containing keys/folders).
Use consul kv with a command like the following to get the same result:
»Total count for all enabled secrets engines
You can get a nicely sorted count per enabled secrets engine with consul kv like so:
This produces results like the following:
These results are somewhat atypical in their simplicity, but you could expect additional UUIDs and their key counts in a more densely configured Vault.
»Total count for a single secrets engine
If you'd prefer a count of only a specific secrets engine, you can alter the URL in your API query. For example, here is the command to grab the count of secrets in the Cubbyhole Secrets Engine only:
Example output:
The Cubbyhole secrets engine with the UUID a54a02ec-e99c-bca7-248d-479a410159b8 contains just the one secret.
You can also get a count per secrets engine with consul kv like this:
The output matches the API call output as expected.
»Token and accessor data
Active tokens and their accessors can be found under vault/sys/token.
This a an example tree of paths from a minimal Vault instance for purposes of illustration:
You can get the total count of all active tokens in Vault like this:
The output is the key count:
or with consul kv:
Likewise, you can get count of token accessors this way:
The output is the key count:
or with consul kv:
»Lease data
The root path for Vault lease data in Consul is at vault/sys/expire/.
Here is a an example tree of paths from a minimal Vault instance for purposes of illustration:
This represents all valid leases in Vault which can be eventually expired, and is broken down further by different auth methods and role paths. The previous example shows leases for the root namespace including credentials from the Token Auth Method, the AppRole Auth Method, and MongoDB Database Secrets Engine.
You can obtain a total lease count with:
The output is the key count:
or with consul kv:
NOTE: This does not represent leases which have actually expired, only those that are pending an eventual expiration. This count can be close to but not typically identical to the active tokens count.
If you want a count for a specific auth method, such as AppRole, use its path instead:
The output is the key count:
or with consul kv:
»Role path
If you're interested in the number of active leases associated with a given role, you can use the role's path.
For example, with the Database Secrets Engines, let's say you have the following roles defined:
- mongodb-readonly
- mysql-readonly
- postgresql-readonly
The roles will be present by name in the Consul key/value store at the following path:
You can count leases in this path with the Consul HTTP API like this:
The output is the key count:
or with consul kv:
The output is the key count:
The output shows that we have 8658 leases under the mongodb-readonly role at the moment.
»What about Enterprise Namespaces?
Beginning with Vault Enterprise version 0.11.0, the concept of Namespaces was introduced.
This changes the previous procedures slightly, in that each namespace will encapsulate its own leases and tokens in paths under the namespace internal storage path name, which you should note is not the same as its user-configured name.
This an example tree of paths from a minimal Vault instance for purposes of illustration:
The actual user-configured namespace name is example-namespace, but this is stored internally as a short unique identifier instead; in the case of our above example, this is value 3E5Y0.
Once you have determined the storage path for the namespace, you can then compose similar commands as those previously shown against the root namespace with your namespaces.
For example, to determine the lease count for all of the AppRole leases in the example-namespace/3E5Y0 namespace with cURL, you must specify the user-specified namespace as the header X-Vault-Namespace value:
The output is the key count:
or with consul kv with the namespace storage path name:
The output is the key count:
Likewise for a count of active tokens:
The output is the key count:
or with consul kv and the namespace storage path name:
The output is the key count:
»Size of data on disk
Consul provides a handy way to get the size of all Vault secrets on disk in the consul kv export command.
You can export the path corresponding to the keys (vault/logical/ by default) with a command like this:
Then you can inspect the size of the resulting JSON file for an approximate idea of the size of your secrets on disk:
There are currently 1000 secrets plus their containing key for the KV (formerly "Generic") secret backend. You can use jq and ls to show the secret count and approximate size on disk with this command:
which results in:
The output shows that my 1000 total secrets occupy approximately 204KB on disk.
You can also use the techniques shown above, to narrow your consul kv export to a particular secret backend, thereby measuring only those secrets instead of all secrets in total.
This technique can also be used for tokens by exporting the proper token key space and measuring the resulting file size, too:
Retrieve the KV pairs at the vault/sys/token/id/ path and save the results in a file, tokens.json.
The following command uses jq to calculate the number of records in the tokens.json.
Keep in mind that namespaces will alter the above paths. If you are unclear about this, please review the What about Enterprise Namespaces? section for more details.
»Summary
In this article we learned about how Vault stores its operational data along with techniques for gathering basic information about them with respect to counts and sizes. You should be able to use what you have learned here as a starting point in inspecting and measuring the characteristics most important to you about your Vault data.
