Cubbyhole Response Wrapping | Vault

The term cubbyhole comes from an Americanism where you get a "locker" or "safe place" to store your belongings or valuables. In Vault, the cubbyhole is your "locker". All secrets are namespaced under your token. If that token expires or is revoked, all the secrets in its cubbyhole are revoked as well.

It is not possible to reach into another token's cubbyhole even as the root user. This is an important difference between the cubbyhole and the key/value secrets engine. The secrets in the key/value secrets engine are accessible to any token for as long as its policy allows it.

»Personas

The end-to-end scenario described in this tutorial involves two personas:

admin with privileged permissions to create tokens
apps trusted entity retrieving secrets from Vault

»Challenge

In order to tightly manage the secrets, you set the scope of who can do what using the Vault policy and attach that to tokens, roles, entities, etc.

Think of a case where you have a trusted entity (Chef, Jenkins, etc.) which reads secrets from Vault. This trusted entity must obtain a token. If the trusted entity or its host machine was rebooted, it must re-authenticate with Vault using a valid token.

How can you securely distribute the initial token to the trusted entity?

»Solution

Use Vault's cubbyhole response wrapping where the initial token is stored in the cubbyhole secrets engine. The wrapped secret can be unwrapped using the single-use wrapping token. Even the user or the system created the initial token won't see the original value. The wrapping token is short-lived and can be revoked just like any other tokens so that the risk of unauthorized access can be minimized.

What is cubbyhole response wrapping?

When response wrapping is requested, Vault creates a temporary single-use token (wrapping token) and insert the response into the token's cubbyhole with a short TTL
Only the expecting client who has the wrapping token can unwrap this secret
Any Vault response can be distributed using the response wrapping

Benefits of using the response wrapping:

It provides cover by ensuring that the value being transmitted across the wire is not the actual secret. It's a reference to the secret.
It provides malfeasance detection by ensuring that only a single party can ever unwrap the token and see what's inside
It limits the lifetime of the secret exposure
- The TTL of the response-wrapping token is separate from the wrapped secret's lease TTL

»Prerequisites

To perform the tasks described in this tutorial, you need to have a Vault environment. Refer to the Getting Started tutorial to install Vault. Make sure that your Vault server has been initialized and unsealed.

NOTE: An interactive tutorial is also available if you do not have a Vault environment to perform the steps described in this tutorial. Click the Show Terminal button to start.

»Policy requirements

NOTE: For the purpose of this tutorial, you can use root token to work with Vault. However, it is recommended that root tokens are only used for just enough initial setup or in emergencies. As a best practice, use tokens with appropriate set of policies based on your role in the organization.

To perform all tasks demonstrated in this tutorial, your policy must include the following permissions:

# Manage tokens
path "auth/token/*" {
  capabilities = [ "create", "read", "update", "delete", "sudo" ]
}

# Write ACL policies
path "sys/policies/acl/*" {
  capabilities = [ "create", "read", "update", "delete", "list" ]
}

# Manage secret/dev secrets engine - for Verification test
path "secret/dev" {
  capabilities = [ "create", "read", "update", "delete", "list" ]
}

# Manage tokenspath "auth/token/*" {  capabilities = [ "create", "read", "update", "delete", "sudo" ]}
# Write ACL policiespath "sys/policies/acl/*" {  capabilities = [ "create", "read", "update", "delete", "list" ]}
# Manage secret/dev secrets engine - for Verification testpath "secret/dev" {  capabilities = [ "create", "read", "update", "delete", "list" ]}

If you are not familiar with policies, complete the policies tutorial.

»Scenario Introduction

Think of a scenario where apps read secrets from Vault. The apps need:

Policy granting "read" permission on the specific path (secret/dev)
Valid tokens to interact with Vault

Response Wrapping Scenario

Setting the appropriate policies and token generation are done by the admin persona. For the admin to distribute the initial token to the app securely, it uses cubbyhole response wrapping. In this tutorial, you perform the following:

Create and wrap a token
Unwrap the secret

NOTE: This tutorial demonstrates how the response wrapping works. To learn more about reading and writing secrets in Vault, refer to the Static Secret tutorial.

»Step 1: Create and wrap a token

(Persona: admin)

When a newly created token is wrapped, Vault inserts the generated token into the cubbyhole of a single-use token, returning that single-use wrapping token. Retrieving the secret requires an unwrap operation against this wrapping token.

In this scenario, an admin user creates a token using response wrapping. To perform the steps in this tutorial, first create a policy for the app.

apps-policy.hcl:

# For testing, read-only on secret/dev path
path "secret/data/dev" {
  capabilities = [ "read" ]
}

# For testing, read-only on secret/dev pathpath "secret/data/dev" {  capabilities = [ "read" ]}

First create an apps policy.

$ vault policy write apps apps-policy.hcl
Policy 'apps' written.

$ vault policy write apps apps-policy.hclPolicy 'apps' written.

Enable key/value v2 secrets engine at secret/ if it's not enabled already.

$ vault secrets enable -path=secret kv-v2

$ vault secrets enable -path=secret kv-v2

Write some test data at secret/dev.

$ vault kv put secret/dev username="webapp" password="my-long-password"

$ vault kv put secret/dev username="webapp" password="my-long-password"

Generate a token for apps persona using response wrapping with TTL of 120 seconds. Also, the generated token should have the apps policy attached. Use the -wrap-ttl flag to specify that the response should be wrapped and the wrap TTL indicates the life of the wrapping token which can be either an integer number of seconds or a string duration of minutes (2m) or hours (1h).

$ vault token create -policy=apps -wrap-ttl=120

Key                              Value
---                              -----
wrapping_token:                  s.3Kf3Xfn58Asr3bSDkRXATHrw
wrapping_accessor:               xzW2I9CMqcALsllhYvqtlsvq
wrapping_token_ttl:              2m
wrapping_token_creation_time:    2019-12-17 07:45:42.537057 -0800 PST
wrapping_token_creation_path:    auth/token/create
wrapped_accessor:                Bh57rT8zuhspG9APjXpGpiAJ

$ vault token create -policy=apps -wrap-ttl=120
Key                              Value---                              -----wrapping_token:                  s.3Kf3Xfn58Asr3bSDkRXATHrwwrapping_accessor:               xzW2I9CMqcALsllhYvqtlsvqwrapping_token_ttl:              2mwrapping_token_creation_time:    2019-12-17 07:45:42.537057 -0800 PSTwrapping_token_creation_path:    auth/token/createwrapped_accessor:                Bh57rT8zuhspG9APjXpGpiAJ

The response is the wrapping token; therefore the admin user does not even see the generated token from the token create command.

»Step 2: Unwrap the secret

(Persona: apps)

The apps persona receives a wrapping token from the admin. In order for the apps to acquire a valid token to read secrets from secret/dev path, it must run the unwrap operation using this token.

NOTE: If a client has been expecting delivery of a response-wrapping token and none arrives, this may be due to an attacker intercepting the token and then preventing it from traveling further. This should cause an alert to trigger an immediate investigation.

The following tasks will be performed to demonstrate the client operations:

Create a token with default policy
Authenticate with Vault using this default token (less privileged token)
Unwrap the secret to obtain more privileged token (apps persona token)
Verify that you can read secret/dev using the appstoken

First, create a token with default policy.

$ vault token create -policy=default
Key                  Value
---                  -----
token                s.VR3mwdk9miJ6VRLFbQe0Duwq
token_accessor       vJOhaSjkwaBhlvwPGR5qTNYc
token_duration       768h
token_renewable      true
token_policies       ["default"]
identity_policies    []
policies             ["default"]

$ vault token create -policy=defaultKey                  Value---                  -----token                s.VR3mwdk9miJ6VRLFbQe0Duwqtoken_accessor       vJOhaSjkwaBhlvwPGR5qTNYctoken_duration       768htoken_renewable      truetoken_policies       ["default"]identity_policies    []policies             ["default"]

Authenticate using the generated token.

$ vault login s.VR3mwdk9miJ6VRLFbQe0Duwq
Success! You are now authenticated. The token information displayed below
is already stored in the token helper. You do NOT need to run "vault login"
again. Future Vault requests will automatically use this token.

Key                  Value
---                  -----
token                s.VR3mwdk9miJ6VRLFbQe0Duwq
token_accessor       vJOhaSjkwaBhlvwPGR5qTNYc
token_duration       767h59m43s
token_renewable      true
token_policies       ["default"]
identity_policies    []
policies             ["default"]

$ vault login s.VR3mwdk9miJ6VRLFbQe0DuwqSuccess! You are now authenticated. The token information displayed belowis already stored in the token helper. You do NOT need to run "vault login"again. Future Vault requests will automatically use this token.
Key                  Value---                  -----token                s.VR3mwdk9miJ6VRLFbQe0Duwqtoken_accessor       vJOhaSjkwaBhlvwPGR5qTNYctoken_duration       767h59m43stoken_renewable      truetoken_policies       ["default"]identity_policies    []policies             ["default"]

Unwrap the secret by passing the wrapping token you received in Step 1.

$ VAULT_TOKEN="s.3Kf3Xfn58Asr3bSDkRXATHrw" vault unwrap

Key                  Value
---                  -----
token                s.e5XHvU78iR08vAUk4Wq9JOBk
token_accessor       zvP8uMEVoe9xS6qHQTXpKekJ
token_duration       768h
token_renewable      true
token_policies       ["apps" "default"]
identity_policies    []
policies             ["apps" "default"]

$ VAULT_TOKEN="s.3Kf3Xfn58Asr3bSDkRXATHrw" vault unwrap
Key                  Value---                  -----token                s.e5XHvU78iR08vAUk4Wq9JOBktoken_accessor       zvP8uMEVoe9xS6qHQTXpKekJtoken_duration       768htoken_renewable      truetoken_policies       ["apps" "default"]identity_policies    []policies             ["apps" "default"]

Verify that this token has apps policy attached.

NOTE: Remember that the wrapping TTL is set to 120 seconds (2 minutes). If you received the wrapping token is not valid or does not exist error, repeat Step 1 to create a token with response wrapping again.

To unwrap the wrapped secret, you can do one of the following.

$ vault unwrap <WRAPPING_TOKEN>

$ vault unwrap <WRAPPING_TOKEN>

$ VAULT_TOKEN=<WRAPPING_TOKEN> vault unwrap

$ VAULT_TOKEN=<WRAPPING_TOKEN> vault unwrap

$ vault login <WRAPPING_TOKEN>
$ vault unwrap

$ vault login <WRAPPING_TOKEN>$ vault unwrap

Once the client acquired the token, future requests can be made using this token.

Example:

$ vault login s.e5XHvU78iR08vAUk4Wq9JOBk

$ vault login s.e5XHvU78iR08vAUk4Wq9JOBk

Verify to see if the token can access secret/dev.

$ vault kv get secret/dev

====== Metadata ======
Key              Value
---              -----
created_time     2019-12-17T19:44:08.220996Z
deletion_time    n/a
destroyed        false
version          1

====== Data ======
Key         Value
---         -----
password    my-long-password
username    webapp

$ vault kv get secret/dev
====== Metadata ======Key              Value---              -----created_time     2019-12-17T19:44:08.220996Zdeletion_time    n/adestroyed        falseversion          1
====== Data ======Key         Value---         -----password    my-long-passwordusername    webapp

»Additional Discussion: Cubbyhole

The cubbyhole secrets engine provides your own private secret storage space where no one else can read (including root). This comes handy when you want to store a password tied to your username that should not be shared with anyone.

The cubbyhole secrets engine is mounted at the cubbyhole/ prefix by default. The secrets you store in the cubbyhole/ path are tied to your token and all tokens are permitted to read and write to the cubbyhole secrets engine by the default policy.

$ vault policy read default

...
# Allow a token to manage its own cubbyhole
path "cubbyhole/*" {
    capabilities = ["create", "read", "update", "delete", "list"]
}
...

$ vault policy read default
...# Allow a token to manage its own cubbyholepath "cubbyhole/*" {    capabilities = ["create", "read", "update", "delete", "list"]}...

To test the cubbyhole secrets engine, perform the following steps. (NOTE: Keep using the apps token from Step 2 to ensure that you are logged in with non-root token.)

Commands to write secrets to the cubbyhole secrets engine:

$ vault write cubbyhole/<PATH> <KEY>=<VALUE>

$ vault write cubbyhole/<PATH> <KEY>=<VALUE>

Commands to read secrets to the cubbyhole secrets engine:

$ vault read cubbyhole/<PATH>

$ vault read cubbyhole/<PATH>

Example:

Write some test secrets under cubbyhole/private/ path.

$ vault write cubbyhole/private/access-token token="123456789abcdefg87654321"
Success! Data written to: cubbyhole/private/access-token

$ vault write cubbyhole/private/access-token token="123456789abcdefg87654321"Success! Data written to: cubbyhole/private/access-token

Now, read the values from cubbyhole/private/access-token.

$ vault read cubbyhole/private/access-token
Key     Value
---     -----
token   123456789abcdefg87654321

$ vault read cubbyhole/private/access-tokenKey     Value---     -----token   123456789abcdefg87654321

Now, try to access the secret using the root token. It should NOT return the secret.

$ VAULT_TOKEN=<ROOT_TOKEN> vault read cubbyhole/private/access-token

No value found at cubbyhole/private/access-token

$ VAULT_TOKEN=<ROOT_TOKEN> vault read cubbyhole/private/access-token
No value found at cubbyhole/private/access-token

Infrastructure

Security

Networking

Applications