April 6 & 7
Learn about Vault, Consul, & more at HashiDays Sydney in Australia Register Now

Monitoring & Troubleshooting

Inspecting Data in Consul Storage

Vault creates and persists critical operational data, such as configuration, leases, roles, secrets, and write ahead logs to its storage. Gathering key facts about these operational data can often be extremely helpful when engaged in advanced troubleshooting.

Vault stores operational data as key/value pairs in a tree like hierarchy that can be visualized and thought of in a similar manner as an OS filesystem with respect to data organization.

Certain techniques used for inspecting and counting the data (such as recursive listing) are not currently possible solely within the context of the Vault HTTP API. With that in mind, this guide uses examples that gather details directly from the the underlying storage instead.

The examples provided here are directly applicable to the Consul storage backend, and include flavors for both the Consul HTTP API and consul CLI.

Notes and prerequisites

Please pay attention to the following list of important notes and prerequisites before you begin.

  1. This is not meant to be a comprehensive guide covering every possible type of data which could be found in Vault; rather, it is a means to help you get started in inspecting your Vault data in Consul.
  2. Please be careful with the commands shown in this guide; to list all of the keys in a particular key/value path can be resource intensive on Consul, so do think about the impact your measuring can have, especially if using these techniques to inspect data a production Consul cluster.
  3. To inspect Vault data in a Consul storage backend you can use the Consul HTTP API with tools like cURL and jq to fetch the information and process it. If it is available, you can also use a consul binary and consul kv commands instead of the Consul HTTP API. It is fine to use an open source consul binary for the command line interactions in this guide.
  4. This guide uses examples from the root namespace; for details about inspecting data within other namespaces created using the Enterprise Namespaces feature, please see the What about Enterprise Namespaces? section.
  5. The Consul server used requires an ACL token and uses TLS so URLs use the https protocol. The following environment variables are set as a convenience so that additional command flags are not required.
    • CONSUL_HTTP_ADDR: This value is the address+port portion (i.e. 127.0.0.1:8500) of the full URL only.
    • CONSUL_HTTP_TOKEN: This value is an ACL token that provides suitable read capabilities for the Consul server cluster's key/value store
    • Here is an example of exporting those environment variables in the terminal session that we will use for the example commands.
      $ export CONSUL_HTTP_ADDR=consul.tacobot.local:8500 \
             CONSUL_HTTP_TOKEN=b4c0ffee-3b77-04af-36d6-738b697872e6
      

New Vault server data example

Let's get to know some Vault data by viewing an actual example with descriptions of each element. When Vault is first initialized and unsealed the persisted data will resemble this example:

├── core
│   ├── _audit
│   ├── _auth
│   ├── _keyring
│   ├── _local-audit
│   ├── _local-auth
│   ├── _local-mounts
│   ├── _master
│   ├── _mounts
│   ├── _seal-config
│   ├── _shamir-kek
│   ├── cluster
│   │   └── local
│   │       └── _info
│   ├── hsm
│   │   └── _barrier-unseal-keys
│   └── wrapping
│       └── _jwtkey
├── index
│   ├── _checkpoint
│   └── pages
│       ├── _04
│       ├── _0f
│       ├── _10
│       ├── _a2
│       ├── _a9
│       ├── _b3
│       ├── _f2
│       └── _fe
├── index-dr
│   ├── _checkpoint
│   └── pages
│       ├── _1a
│       ├── _22
│       ├── _58
│       ├── _88
│       ├── _d1
│       ├── _e1
│       └── _e4
├── logical
│   └── a7f1489c-7e22-06e6-6c88-37fafe32f4f1
│       └── _casesensitivity
├── sys
│   ├── policy
│   │   ├── _control-group
│   │   ├── _default
│   │   └── _response-wrapping
│   └── token
│       ├── _salt
│       ├── accessor
│       │   └── _e2c12bf2ce316b10dc60ec173b53aaff8fc402ca
│       └── id
│           └── _h322cd680d8ee8b202dd8ffdc110d0c212022d57934b6e7aa65f6b869b27a0ec4
└── wal
    └── logs
        └── 00000000
            ├── _0001
            ├── _0002
            ├── _0003
            ├── _0004
            ├── _0005
            ├── _0006
            ├── _0007
            ├── _0008
            ├── _0009
            ├── _000a
            ├── _000b
            ├── _000c
            ├── _000d
            ├── _000e
            ├── _000f
            └── _0010

A total of 73 key/value pairs are present in this example, representing all of the data necessary for Vault to begin operations. A Vault server that is in production will have considerably more data and key/value pairs related to its specific auth methods, secrets engines, and so on.

Here is a brief explanation of each major branch and the elements within them from example.

  • core: Items contained in core are critical and internal to Vault operations; these include data about internal auditing, authentication, keyring, mounts, the master key, the seal configuration, cluster information, HSM barrier unseal keys, seal wrapping, and more.
  • index: This is local index data which can be used by the Performance Standby feature
  • index-dr: This is index data for the Disaster Recovery mode of Vault Enterprise Replication
  • logical: Dynamic secret configuration and static secrets are found here
  • sys: System data includes policy configuration along with tokens and their accessors.
  • wal: Write ahead logs (WAL) are present in Vault Enterprise installations to support the Performance Standby feature and assist with enabling Enterprise Replication

Those are the basics for now.

We will continue by inspecting secrets engine data.

Secret engine data

A common question about Vault secret data during support and operations troubleshooting scenarios is How many secrets do we currently have in Vault for this secrets engine?

To begin down the path of answering this question, let's develop some understanding of the secrets engine data storage structure with further examples.

Vault stores secrets engine data in the configured storage at the default path /vault/logical/<UUID>/ where <UUID> represents a unique identifier for the specific enabled secret backend.

When a new Vault is initialized and unsealed, only the Identity secrets engine is configured and present in the storage as shown in the example data:

├── logical
│   └── a7f1489c-7e22-06e6-6c88-37fafe32f4f1
│       └── _casesensitivity

After Vault is further configured, the logical path would be expected to contain more secrets engine data. For example, here is a tree view of example secret engine data with detailed explanation of each element.

├── logical
│   └── e2b7c3e2-3e21-3391-b73c-8a991a65789d
│       └── f030471f-1f42-6c61-9d42-179427741f49
│           ├── _salt
│           ├── _upgrading
│           ├── archive
│           │   └── _metadata
│           ├── metadata
│           │   └── _Fz2pkGY3Mo2Umyt7REtEpyNFJwWVrmS54tZbMBfbJDuuYhtcl6Wmgy1Byo7cqI8R3yUNkmtjAfb9Omw4mQJ
│           ├── policy
│           │   └── _metadata
│           └── versions
│               ├── 580
│               │   └── _081e7244b38d5761da22c6958357d27cb28fc31a8d306e356bc371db1021f
│               └── dfd
│                   └── _dca872765e8bb300874869f76994b3b7f5b811ea6103b2367cbe4261d55e4
│   ├── 2788376d-7042-4737-1ebd-9f6391a01f4e
│   │   ├── _ca
│   │   ├── _crl
│   │   ├── _urls
│   │   ├── config
│   │   │   └── _ca_bundle
│   │   └── role
│   │       └── _tacobot-root
│   ├── b7183aba-6e64-e001-fe57-3e7e4508fc0c
│   │   ├── _ca
│   │   ├── _crl
│   │   ├── _urls
│   │   ├── certs
│   │   │   ├── _17-57-81-3f-5f-08-43-00-79-97-b5-0c-b3-0e-5e-cd-49-a5-88-21
│   │   │   └── _19-19-9f-17-91-3b-8d-da-77-c2-c2-f9-37-1a-4f-19-4c-5b-f2-9a
│   │   ├── config
│   │   │   └── _ca_bundle
│   │   └── role
│   │       └── _tacobot-int
│   ├── cb1bfb31-3ccb-ef29-6352-874902c3a021
│   │   ├── config
│   │   │   ├── _mongodb
│   │   │   └── _mysql
│   │   └── role
│   │       ├── _tacobot-mongodb-readonly
│   │       └── _tacobot-mysql-readonly
│   ├── d1689597-4f78-a30b-7532-e7806be9fcba
│   │   └── _casesensitivity
│   └── fbd73ad9-4f9c-45be-5be2-3758d04808af
│       └── 9t7pwHwrPD0yiGuLKMHi912x
│           └── _my-secret

The previous example shows paths for several secrets engines in the root namespace; here are details on each secrets engine and its associated elements:

  • e2b7c3e2-3e21-3391-b73c-8a991a65789d A KV Secrets Engine - Version 2 containing internal configuration and metadata along with the secret data versions found under the versions key.
  • 2788376d-7042-4737-1ebd-9f6391a01f4e A PKI secrets engine which represents the root Certificate Authority (CA). It contains the CA information, The Certificate Revocation List (CRL) data, the URL configuration, internal configuration (with a CA bundle), and a role in this case called tacobot-root.
  • b7183aba-6e64-e001-fe57-3e7e4508fc0c A PKI secrets engine which represents the intermediate CA. It contains the CA information, The CRL data, the URL configuration, internal configuration (with a CA bundle), and a role in this case called tacobot-int. Note also that it has a certs key with some certificate serial numbers present which represent the certificates issued from the tacobot-int role.
  • cb1bfb31-3ccb-ef29-6352-874902c3a021 A Database Secrets Engine with configuration and roles for MongoDB and MySQL
  • d1689597-4f78-a30b-7532-e7806be9fcba An Identity Secrets Engine is the identity management solution for Vault and enabled by default. This secrets engine cannot be disabled or moved.
  • fbd73ad9-4f9c-45be-5be2-3758d04808af A Cubbyhole Secrets Engine which is enabled by default. It cannot be disabled, moved, or enabled multiple times.

Total secret count

You can get an approximate total count of keys (this includes secrets and their containing keys) across all currently enabled secrets engines with a command like this:

$ curl -s https://$CONSUL_HTTP_ADDR/v1/kv/vault/logical/\?keys \
       -H "X-Consul-Token: $CONSUL_HTTP_TOKEN" | \
       jq '. | length'

The output that results should be just the count, for example:

42001

The output shows that these Vault data contain approximately 40,000 static secrets (and their containing keys/folders).

Use consul kv with a command like the following to get the same result:

$ consul kv get -recurse -separator="" -keys vault/logical/ | wc -l
42001

Total count for all enabled secrets engines

You can get a nicely sorted count per enabled secrets engine with consul kv like so:

$ consul kv get \
  -recurse \
  -keys \
  -separator=" " \
  vault/logical/ \
  | awk -F"/" '{print $3}' | sort | uniq -c | sort -nr

This produces results like the following:

42000 e2b7c3e2-3e21-3391-b73c-8a991a65789d
    1 fbd73ad9-4f9c-45be-5be2-3758d04808af

These results are somewhat atypical in their simplicity, but you could expect additional UUIDs and their key counts in a more densely configured Vault.

Total count for a single secrets engine

If you'd prefer a count of only a specific secrets engine, you can alter the URL in your API query. For example, here is the command to grab the count of secrets in the Cubbyhole Secrets Engine only:

$ curl -s https://$CONSUL_HTTP_ADDR/v1/kv/vault/logical/a54a02ec-e99c-bca7-248d-479a410159b8\?keys \
       -H "X-Consul-Token: $CONSUL_HTTP_TOKEN" | \
       jq '. | length'

Example output:

1

The Cubbyhole secrets engine with the UUID a54a02ec-e99c-bca7-248d-479a410159b8 contains just the one secret.

You can also get a count per secrets engine with consul kv like this:

$ consul kv get \
  -recurse \
  -keys \
  vault/logical/a54a02ec-e99c-bca7-248d-479a410159b8/ | wc -l

The output matches the API call output as expected.

1

Token and accessor data

Active tokens and their accessors can be found under vault/sys/token.

This a an example tree of paths from a minimal Vault instance for purposes of illustration:

sys
└── token
      ├── _salt
      ├── accessor
      │   ├── _0f396549a86c562ab0a8a7d7f81eb7481f75ced6
      │   ├── _2221c2b6f68acdea65f2da7ea68a131f1a5586a6
      │   ├── _24fe37f933b5b32ec7a90c176d216478eaae7df6
      │   ├── _5ca2e944d3f0ba556b5a610dc25fff2df2f9b600
      │   ├── _d482863a44b91786a913d2773170df7db2e92a3c
      │   └── _fd7013db5298d314ae6b95aa100dd29ba43ec203
      ├── id
      │   ├── _h11f74c2a6d66a40b47b70d5ed0a8a9b92bae8e9b6ad3eba466cc51e10ec16c94
      │   ├── _h1e24e12f70cd74d13501fb71029ccc4d248d1722f545a052666e2a46cc8323f6
      │   ├── _h2cd1083faad0ec4dce12a63b2f0637188909c2d25d2b33511f26b9bd716e95ad
      │   ├── _h66cdedfaed4899cb331253963f100ddfbf55e6dfd91fb744d5f01129dd8d1e3f
      │   ├── _h9569ccb2eb6ede2e549e73a8f64a09a629349e46fed867971848bd36a55efad7
      │   └── _h9b70418b656d7b34e270580c9082e2c4d45d688eb35365289b3058138ca7e0f4
      └── parent
          └── h66cdedfaed4899cb331253963f100ddfbf55e6dfd91fb744d5f01129dd8d1e3f
              └── _h1e24e12f70cd74d13501fb71029ccc4d248d1722f545a052666e2a46cc8323f6

You can get the total count of all active tokens in Vault like this:

$ curl -s https://$CONSUL_HTTP_ADDR/v1/kv/vault/sys/token/id/\?keys \
       -H "X-Consul-Token: $CONSUL_HTTP_TOKEN" | \
       jq '. | length'
9001

or with consul kv:

$ consul kv get -recurse -separator="" -keys vault/sys/token/id/ | wc -l
9001

Likewise, you can get count of token accessors this way:

$ curl -s https://$CONSUL_HTTP_ADDR/v1/kv/vault/sys/token/accessor/\?keys \
       -H "X-Consul-Token: $CONSUL_HTTP_TOKEN" | \
       jq '. | length'
9001

or with consul kv:

$ consul kv get -recurse -separator="" -keys vault/sys/token/accessor/ | wc -l
9001

Lease data

The root path for Vault lease data in Consul is at vault/sys/expire/.

Here is a an example tree of paths from a minimal Vault instance for purposes of illustration:

sys
├── expire
│   ├── id
│   │   ├── auth
│   │   │   ├── token
│   │   │   │   └── create
│   │   │   │       ├── _h1e3cc46d6597509414b45c097298d55bcfd22be45798865fef79e00fcf5a34b4
│   │   │   │       └── _ha04379fb15fafefe0c7c8b04dd73bc45b1ca43e94bfe63bb55c6af1a7f0b89e1
│   │   │   └── vaultron-approle
│   │   │       └── login
│   │   │           ├── _h11f74c2a6d66a40b47b70d5ed0a8a9b92bae8e9b6ad3eba466cc51e10ec16c94
│   │   │           ├── _h2cd1083faad0ec4dce12a63b2f0637188909c2d25d2b33511f26b9bd716e95ad
│   │   │           ├── _h9569ccb2eb6ede2e549e73a8f64a09a629349e46fed867971848bd36a55efad7
│   │   │           └── _h9b70418b656d7b34e270580c9082e2c4d45d688eb35365289b3058138ca7e0f4
│   │   └── vaultron-database
│   │       └── creds
│   │           └── mongodb-root-ns-readonly
│   │               ├── _1cvqF35LKbwkYKZnGihsy2bN
│   │               ├── _DD9avdMp60JNwcsAffEzq7Dy
│   │               ├── _fdhBSDWTCbwYoHieME6dn4ZR
│   │               └── _m4Q7d0laIHkkfS9GclJ8URsV
│   └── token
│       └── h66cdedfaed4899cb331253963f100ddfbf55e6dfd91fb744d5f01129dd8d1e3f
│           ├── _25dccca168f94a6037281c9a80c2ebc312bbb976
│           ├── _52b0e614bd3ee6cd556e4bb22d8db191de8e4abc
│           ├── _68203006c10066f7b33bfe97e059d319e663522e
│           ├── _807660937903a6e9fcae1837da6ef2217263d8df
│           ├── _h5ae308eab7ecee4878ce9116af3eceb48f7db4ecfb9f0793da20911f002acef1
│           ├── _h927c982ada4cb8a867ef85fc092652e6750c89a9761cddd3c8c9b63118e194a8
│           ├── _hdeb7e7109170c0c66fb556421812e10d3d1793f13255e7f6e26cbbb9a1caa187
│           └── _hfe5e69e03527214bfd3996598c0f64c7fcefcd93f4c14411f5ba6e2245ced183

This represents all valid leases in Vault which can be eventually expired, and is broken down further by different auth methods and role paths. The previous example shows leases for the root namespace including credentials from the Token Auth Method, the AppRole Auth Method, and MongoDB Database Secrets Engine.

You can obtain a total lease count with:

$ curl -s https://$CONSUL_HTTP_ADDR/v1/kv/vault/sys/expire/id/auth/\?keys \
       -H "X-Consul-Token: $CONSUL_HTTP_TOKEN" | \
       jq '. | length'
40238

or with consul kv:

$ consul kv get -recurse -separator="" -keys vault/sys/expire/id/auth/ | wc -l
40238

If you want a count for another auth method, such as AppRole, use its path instead:

$ curl -s https://$CONSUL_HTTP_ADDR/v1/kv/vault/sys/expire/id/auth/approle/login/\?keys \
       -H "X-Consul-Token: $CONSUL_HTTP_TOKEN" | \
       jq '. | length'
114598

or with consul kv:

$ consul kv get \
    -recurse \
    -separator="" \
    -keys vault/sys/expire/id/auth/approle/login/ \
    | wc -l
114598

Role path

If you're interested in the number of active leases associated with a given role, you can use the role's path.

For example, with the Database Secrets Engines, let's say you have the following roles defined:

  • mongodb-readonly
  • mysql-readonly
  • postgresql-readonly

The roles will be present by name in the Consul key/value store at the following path:

/v1/kv/vault/sys/expire/id/database/creds/

You can count leases in this path with the Consul HTTP API like this:

$ curl -s https://$CONSUL_HTTP_ADDR/v1/kv/vault/sys/expire/id/database/creds/mongodb-readonly/\?keys \
    -H "X-Consul-Token: $CONSUL_HTTP_TOKEN" | \
    jq '. | length'
8658

or with consul kv:

$ consul kv get \
    -recurse \
    -separator="" \
    -keys vault/sys/expire/id/database/creds/mongodb-readonly/ \
    | wc -l
8658

The output shows that we have 8658 leases under the mongodb-readonly role at the moment.

What about Enterprise Namespaces?

Beginning with Vault Enterprise version 0.11.0, the concept of Namespaces was introduced.

This changes the previous procedures slightly, in that each namespace will encapsulate its own leases and tokens in paths under the namespace internal storage path name, which you should note is not the same as its user-configured name.

This an example tree of paths from a minimal Vault instance for purposes of illustration:

├── namespaces
│   ├── 3E5Y0
│   │   └── sys
│   │       ├── expire
│   │       │   └── id
│   │       │       ├── auth
│   │       │       │   └── approle
│   │       │       │       └── login
│   │       │       │           ├── _h77ec2e32d144d6d75873a8b2098996b14ee70ed18addf6a4f5dd99d678d28b0b.3E5Y0
│   │       │       │           ├── _h8fba78bf0d4e497eb662709bf9e0f1e1ae0b053ec7cb3dbd3bc45ec76c66b693.3E5Y0
│   │       │       │           ├── _hf15a2ad701c9ec86089a5eb4704d2fbbbcff1fe66d118d005107a7dc83fb21ec.3E5Y0
│   │       │       │           └── _hf2b6bbafc5db6125191e09cbc7eaee9a080953217df4a105601c1fbf8b8ce492.3E5Y0
│   │       │       └── database
│   │       │           └── creds
│   │       │               └── mongodb-example-namespace-readonly
│   │       │                   ├── _HoNqW43Gy4N5Mbg3LzOEpuLa.3E5Y0
│   │       │                   ├── _OKHi04a98CLO2CYMMXHwduNj.3E5Y0
│   │       │                   ├── _YbfCjSZKwHUJeoAB31wIwn7f.3E5Y0
│   │       │                   └── _nNcN5KNlzh47JwhmGVMIMvkV.3E5Y0
│   │       ├── policy
│   │       │   ├── _control-group
│   │       │   ├── _default
│   │       │   └── _response-wrapping
│   │       └── token
│   │           ├── _salt
│   │           ├── accessor
│   │           │   ├── _h61e29be9d79c1d597cefab1ae8242fc8dffa6cb72ea1514260f1250632401638
│   │           │   ├── _ha6ddbad7b6237b0de4c03faf9cb82ae87aca4a8912b9c4b09c9f6b00dbe7c32e
│   │           │   ├── _ha9d7d995ddc93c4e136d686fc24ef4ca9f28e4793358b49e68c7e70e13843815
│   │           │   └── _he05dc5ca4f1ba60b20e406d4215d4d469a2a26231fab8f5067a97893147d95ea
│   │           └── id
│   │               ├── _h77ec2e32d144d6d75873a8b2098996b14ee70ed18addf6a4f5dd99d678d28b0b
│   │               ├── _h8fba78bf0d4e497eb662709bf9e0f1e1ae0b053ec7cb3dbd3bc45ec76c66b693
│   │               ├── _hf15a2ad701c9ec86089a5eb4704d2fbbbcff1fe66d118d005107a7dc83fb21ec
│   │               └── _hf2b6bbafc5db6125191e09cbc7eaee9a080953217df4a105601c1fbf8b8ce492
│   └── _info

The actual user-configured namespace name is example-namespace, but this is stored internally as a short unique identifier instead; in the case of our above example, this is value 3E5Y0.

Once you have determined the storage path for the namespace, you can then compose similar commands as those previously shown against the root namespace with your namespaces.

For example, to determine the lease count for all of the AppRole leases in the example-namespace/3E5Y0 namespace with cURL, you must specify the user-specified namespace as the header X-Vault-Namespace value:

$ curl -s \
    https://$CONSUL_HTTP_ADDR/v1/kv/vault/sys/token/accessor/?keys \
    -H "X-Consul-Token: $CONSUL_HTTP_TOKEN" \
    -H "X-Vault-Namespace: example-namespace" \
    | jq '. | length'
19740

or with consul kv with the namespace storage path name:

$ consul kv get \
    -recurse \
    -separator="" \
    -keys vault/namespaces/3E5Y0/sys/expire/id/auth/approle/login/ \
    | wc -l
19740

Likewise for a count of active tokens:

$ curl -s \
    $CONSUL_HTTP_ADDR/v1/kv/vault/sys/token/id/?keys \
    -H "X-Consul-Token: $CONSUL_HTTP_TOKEN" \
    -H "X-Vault-Namespace: example-namespace" \
    | jq '. | length'
19742

or with consul kv and the namespace storage path name:

$ consul kv get \
    -recurse -separator="" \
    -keys vault/namespaces/3E5Y0/sys/token/id/ \
    | wc -l
19742

Size of data on disk

Consul provides a handy way to get the size of all Vault secrets on disk in the consul kv export command.

You can export the path corresponding to the keys (vault/logical/ by default) with a command like this:

$ consul kv export vault/logical/ > vault_secrets.json

Then you can inspect the size of the resulting JSON file for an approximate idea of the size of your secrets on disk:

$ ls -lh vault_secrets.json
-rw-r--r--  1 consul  consul   204K Oct 20 09:27 vault_secrets.json

There are currently 1000 secrets plus their containing key for the KV (formerly "Generic") secret backend. You can use jq and ls to show the secret count and approximate size on disk with this command:

$ JSON=vault_secrets.json ; \
echo $JSON: `jq '. | length' $JSON ; ls -lh $JSON | awk '{print $5}'`

which results in:

vault_secrets.json: 1001 204K

The output shows that my 1000 total secrets occupy approximately 204KB on disk.

You can also use the techniques shown above, to narrow your consul kv export to a particular secret backend, thereby measuring only those secrets instead of all secrets in total.

This technique can also be used for tokens by exporting the proper token key space and measuring the resulting file size, too:

$ consul kv export vault/sys/token/id/ > tokens.json
$ JSON=tokens.json ; \
echo $JSON: `jq '. | length' $JSON ; ls -lh $JSON | awk '{print $5}'`
tokens.json: 501 334K

Summary

In this article we learned about how Vault stores its operational data along with techniques for gathering basic information about them with respect to counts and sizes. You should be able to use what you have learned here as a starting point in inspecting and measuring the characteristics most important to you about your Vault data.

Help and Reference

  1. Consul KV store HTTP API
  2. consul kv CLI
  3. Key/Value Secrets Engine
  4. Cubbyhole Secrets Engine
  5. Databases Secrets Engine
  6. MongoDB Database Secrets Engine
  7. AppRole Auth Method
  8. cURL
  9. jq
  10. consul kv
  11. Namespaces