April 6 & 7
Learn about Vault, Consul, & more at HashiDays Sydney in Australia Register Now

Secrets Management

Cubbyhole Response Wrapping

The term cubbyhole comes from an Americanism where you get a "locker" or "safe place" to store your belongings or valuables. In Vault, the cubbyhole is your "locker". All secrets are namespaced under your token. If that token expires or is revoked, all the secrets in its cubbyhole are revoked as well.

It is not possible to reach into another token's cubbyhole even as the root user. This is an important difference between the cubbyhole and the key/value secrets engine. The secrets in the key/value secrets engine are accessible to any token for as long as its policy allows it.

Personas

The end-to-end scenario described in this guide involves two personas:

  • admin with privileged permissions to create tokens
  • apps trusted entity retrieving secrets from Vault

Challenge

In order to tightly manage the secrets, you set the scope of who can do what using the Vault policy and attach that to tokens, roles, entities, etc.

Think of a case where you have a trusted entity (Chef, Jenkins, etc.) which reads secrets from Vault. This trusted entity must obtain a token. If the trusted entity or its host machine was rebooted, it must re-authenticate with Vault using a valid token.

How can you securely distribute the initial token to the trusted entity?

Solution

Use Vault's cubbyhole response wrapping where the initial token is stored in the cubbyhole secrets engine. The wrapped secret can be unwrapped using the single-use wrapping token. Even the user or the system created the initial token won't see the original value. The wrapping token is short-lived and can be revoked just like any other tokens so that the risk of unauthorized access can be minimized.

What is cubbyhole response wrapping?

  • When response wrapping is requested, Vault creates a temporary single-use token (wrapping token) and insert the response into the token's cubbyhole with a short TTL
  • Only the expecting client who has the wrapping token can unwrap this secret
  • Any Vault response can be distributed using the response wrapping

Benefits of using the response wrapping:

  • It provides cover by ensuring that the value being transmitted across the wire is not the actual secret. It's a reference to the secret.
  • It provides malfeasance detection by ensuring that only a single party can ever unwrap the token and see what's inside
  • It limits the lifetime of the secret exposure
    • The TTL of the response-wrapping token is separate from the wrapped secret's lease TTL

Prerequisites

To perform the tasks described in this guide, you need to have a Vault environment. Refer to the Getting Started guide to install Vault. Make sure that your Vault server has been initialized and unsealed.

NOTE: An interactive tutorial is also available if you do not have a Vault environment to perform the steps described in this guide. Click the Show Tutorial button to launch the tutorial.

Policy requirements

To perform all tasks demonstrated in this guide, your policy must include the following permissions:

# Manage tokens
path "auth/token/*" {
  capabilities = [ "create", "read", "update", "delete", "sudo" ]
}

# Write ACL policies
path "sys/policies/acl/*" {
  capabilities = [ "create", "read", "update", "delete", "list" ]
}

# Manage secret/dev secrets engine - for Verification test
path "secret/dev" {
  capabilities = [ "create", "read", "update", "delete", "list" ]
}

If you are not familiar with policies, complete the policies guide.

Scenario Introduction

Think of a scenario where apps read secrets from Vault. The apps need:

  • Policy granting "read" permission on the specific path (secret/dev)
  • Valid tokens to interact with Vault

Response Wrapping Scenario

Setting the appropriate policies and token generation are done by the admin persona. For the admin to distribute the initial token to the app securely, it uses cubbyhole response wrapping. In this guide, you perform the following:

  1. Create and wrap a token
  2. Unwrap the secret

Step 1: Create and wrap a token

(Persona: admin)

When a newly created token is wrapped, Vault inserts the generated token into the cubbyhole of a single-use token, returning that single-use wrapping token. Retrieving the secret requires an unwrap operation against this wrapping token.

In this scenario, an admin user creates a token using response wrapping. To perform the steps in this guide, first create a policy for the app.

apps-policy.hcl:

# For testing, read-only on secret/dev path
path "secret/data/dev" {
  capabilities = [ "read" ]
}

CLI command / API call using cURL

CLI command

  1. First create an apps policy.

    $ vault policy write apps apps-policy.hcl
    Policy 'apps' written.
    
  2. Enable key/value v2 secrets engine at secret/ if it's not enabled already.

    $ vault secrets enable -path=secret kv-v2
    
  3. Write some test data at secret/dev.

    $ vault kv put secret/dev username="webapp" password="my-long-password"
    
  4. Generate a token for apps persona using response wrapping with TTL of 120 seconds. Also, the generated token should have the apps policy attached. Use the -wrap-ttl flag to specify that the response should be wrapped and the wrap TTL indicates the life of the wrapping token which can be either an integer number of seconds or a string duration of minutes (2m) or hours (1h).

    $ vault token create -policy=apps -wrap-ttl=120
    
    Key                              Value
    ---                              -----
    wrapping_token:                  s.9QFJ8mRxGJD0e7kFfFIbdpDM
    wrapping_accessor:               S0zKNUr2ENbnCtj0YyriO31b
    wrapping_token_ttl:              2m
    wrapping_token_creation_time:    2019-12-17 09:45:42.537057 -0800 PST
    wrapping_token_creation_path:    auth/token/create
    wrapped_accessor:                VmBKXoc19ZLZlHGl0nQCvV6r
    

    The response is the wrapping token; therefore the admin user does not even see the generated token from the token create command.

API call using cURL

  1. First create an apps policy using sys/policies/acl endpoint.

    Create an API request payload containing the stringified apps policy.

    $ cat payload.json
    {
      "policy": "path \"secret/data/dev\" { capabilities = [ \"read\" ] }"
    }
    

    Create apps policy.

    $ curl --header "X-Vault-Token: <TOKEN>"\
          --request PUT --data @payload.json \
          http://127.0.0.1:8200/v1/sys/policies/acl/apps
    

    Where <TOKEN> is your valid token, and <PAYLOAD> includes policy name and stringified policy.

  2. Enable key/value v2 secrets engine at secret/ if it's not enabled already.

    $ curl --header "X-Vault-Token: <TOKEN>" \
          --request POST \
          --data '{"type":"kv-v2"}' \
          https://eu-west-1.compute.com:8200/v1/sys/mounts/secret
    
  3. Create an API request payload containing some test data to write at secret/dev.

    $ tee payload.json <<EOF
    {
      "data": {
        "username": "webapp",
        "password": "my-long-password"
      }
    }
    EOF
    
  4. Create some test data at secret/dev.

    $ curl --header "X-Vault-Token: <TOKEN>" \
          --request POST \
          --data @payload.json \
          https://eu-west-1.compute.com:8200/v1/secret/dev
    
  5. Response wrapping is per-request and is triggered by providing to Vault the desired TTL for a response-wrapping token for that request. This is set using the X-Vault-Wrap-TTL header in the request and can be either an integer number of seconds or a string duration.

    $ curl --header "X-Vault-Wrap-TTL: <TTL>" \
          --header "X-Vault-Token: <TOKEN>" \
          --request <HTTP_VERB> \
          --data '<PARAMETERS>' \
          <VAULT_ADDRESS>/v1/<ENDPOINT>
    

    Where <TTL> can be either an integer number of seconds or a string duration of minutes (2m), or hours (2h).

    Generate a token for apps persona using response wrapping with TTL of 120 seconds. Also, attach the apps policy to the generated token.

    $ curl --header "X-Vault-Wrap-TTL: 120" \
          --header "X-Vault-Token: <TOKEN>" \
          --request POST \
          --data '{"policies":["apps"]}' \
          http://127.0.0.1:8200/v1/auth/token/create | jq
    {
      ...
      "wrap_info": {
        "token": "s.3Kf3Xfn58Asr3bSDkRXATHrw",
        "accessor": "xzW2I9CMqcALsllhYvqtlsvq",
        "ttl": 120,
        "creation_time": "2019-12-17T09:49:49.930503-08:00",
        "creation_path": "auth/token/create",
        "wrapped_accessor": "Bh57rT8zuhspG9APjXpGpiAJ"
      },
      ...
    }
    
    $ curl --header "X-Vault-Wrap-TTL: <TTL>" \
          --header "X-Vault-Token: <TOKEN>" \
          --request <HTTP_VERB> \
          --data '<PARAMETERS>' \
          <VAULT_ADDRESS>/v1/<ENDPOINT>
    

    Where <TTL> can be either an integer number of seconds or a string duration of minutes (2m), or hours (2h).

    Generates a token for apps persona using response wrapping with TTL of 120 seconds. Also, attach the apps policy to the generated token.

    $ curl --header "X-Vault-Wrap-TTL: 120" \
          --header "X-Vault-Token: ..." \
          --request POST \
          --data '{"policies":["apps"]}' \
          http://127.0.0.1:8200/v1/auth/token/create | jq
    {
      ...
      "wrap_info": {
        "token": "s.3Kf3Xfn58Asr3bSDkRXATHrw",
        "accessor": "xzW2I9CMqcALsllhYvqtlsvq",
        "ttl": 120,
        "creation_time": "2019-12-17T09:49:49.930503-08:00",
        "creation_path": "auth/token/create",
        "wrapped_accessor": "Bh57rT8zuhspG9APjXpGpiAJ"
      },
      ...
    }
    

    The response is the wrapping token; therefore the admin user does not even see the generated client token.

Step 2: Unwrap the secret

(Persona: apps)

The apps persona receives a wrapping token from the admin. In order for the apps to acquire a valid token to read secrets from secret/dev path, it must run the unwrap operation using this token.

The following tasks will be performed to demonstrate the client operations:

  1. Create a token with default policy
  2. Authenticate with Vault using this default token (less privileged token)
  3. Unwrap the secret to obtain more privileged token (apps persona token)
  4. Verify that you can read secret/dev using the appstoken

CLI command / API call using cURL

CLI command

  1. First, create a token with default policy.

    $ vault token create -policy=default
    Key                  Value
    ---                  -----
    token                s.VR3mwdk9miJ6VRLFbQe0Duwq
    token_accessor       vJOhaSjkwaBhlvwPGR5qTNYc
    token_duration       768h
    token_renewable      true
    token_policies       ["default"]
    identity_policies    []
    policies             ["default"]
    
  2. Authenticate using the generated token.

    $ vault login s.VR3mwdk9miJ6VRLFbQe0Duwq
    Success! You are now authenticated. The token information displayed below
    is already stored in the token helper. You do NOT need to run "vault login"
    again. Future Vault requests will automatically use this token.
    
    Key                  Value
    ---                  -----
    token                s.VR3mwdk9miJ6VRLFbQe0Duwq
    token_accessor       vJOhaSjkwaBhlvwPGR5qTNYc
    token_duration       767h59m43s
    token_renewable      true
    token_policies       ["default"]
    identity_policies    []
    policies             ["default"]
    
  3. Unwrap the secret by passing the wrapping token you received in Step 1.

    $ VAULT_TOKEN="s.3Kf3Xfn58Asr3bSDkRXATHrw" vault unwrap
    
    Key                  Value
    ---                  -----
    token                s.e5XHvU78iR08vAUk4Wq9JOBk
    token_accessor       zvP8uMEVoe9xS6qHQTXpKekJ
    token_duration       768h
    token_renewable      true
    token_policies       ["apps" "default"]
    identity_policies    []
    policies             ["apps" "default"]
    

    Verify that this token has apps policy attached.

    NOTE: Remember that the wrapping TTL is set to 120 seconds (2 minutes). If you received the wrapping token is not valid or does not exist error, repeat Step 1 to create a token with response wrapping again.

    The unwrap the wrapped secret, you can do one of the following.

    $ vault unwrap <WRAPPING_TOKEN>
    

    Or

    $ VAULT_TOKEN=<WRAPPING_TOKEN> vault unwrap
    

    Or

    $ vault login <WRAPPING_TOKEN>
    $ vault unwrap
    
  4. Once the client acquired the token, future requests can be made using this token.

    Example:

    $ vault login s.e5XHvU78iR08vAUk4Wq9JOBk
    

    Verify to see if the token can access secret/dev.

    $ vault kv get secret/dev
    
    ====== Metadata ======
    Key              Value
    ---              -----
    created_time     2019-12-17T19:44:08.220996Z
    deletion_time    n/a
    destroyed        false
    version          1
    
    ====== Data ======
    Key         Value
    ---         -----
    password    my-long-password
    username    webapp
    

API call using cURL

  1. First, create a token with default policy.

    $ curl --header "X-Vault-Token: <TOKEN>" --request POST \
        --data '{"policies": "default"}' \
        http://127.0.0.1:8200/v1/auth/token/create | jq
    {
      ...
      "auth": {
        "client_token": "5fe14760-b0fd-22dc-403c-14a05003b67f",
        "accessor": "e709610e-916e-f7e3-b93b-41f4dfdca7a0",
        "policies": [
          "default"
        ],
        ...
    }
    }
    
  2. Unwrap the secret by passing the wrapping token you received in Step 1 in the request header (X-Vault-Token):

    $ curl --header "X-Vault-Token: s.3Kf3Xfn58Asr3bSDkRXATHrw" \
          --request POST \
          http://127.0.0.1:8200/v1/sys/wrapping/unwrap | jq
    {
      ...
      "auth": {
        "client_token": "s.KqSjapk3yCdQfvkegQWg3XGm",
        "accessor": "DFXxnHQihkGKe3nzBbw2yung",
        "policies": [
          "apps",
          "default"
        ],
        ...
      }
    }
    

    Verify that this token has apps policy attached.

    NOTE: Remember that the wrapping TTL is set to 120 seconds (2 minutes). If you received the wrapping token is not valid or does not exist error, repeat Step 1 to create a token with response wrapping again.

  3. Once the client acquired the token, future requests can be made using this token.

    $ curl --header "X-Vault-Token: s.KqSjapk3yCdQfvkegQWg3XGm" \
          http://127.0.0.1:8200/v1/secret/data/dev | jq
    {
      ...
      "data": {
        "data": {
          "password": "my-long-password",
          "username": "webapp"
        },
        ...
    }
    

Web UI

As of Vault 1.0, the Web UI supports wrapping and unwrapping secrets. The following steps demonstrate the following:

  • A privileged user wraps secrets using response wrapping
  • The receiving client unwraps the secrets

Open a web browser and launch the Vault UI (e.g. http://127.0.0.1:8200/ui) and then login.

  1. Click the Access tab, and select Enable new method.

  2. Select Username & Password and click Next. Auth Method

  3. Accept the default and click Enable Method.

  4. Click the Vault CLI shell icon (>_) to open a command shell. Execute the following command to create a new user, bob for testing: vault write auth/userpass/users/bob password=training Create bob

    NOTE: In absence of the policies argument, the user only has the default policy assigned.

  5. Click the icon (>_) again to hide the shell.

  6. Select Secrets.

  7. Select secret/ from the Secrets Engines list, and then click Create secret.

  8. Enter myapp/config in the Path for this secret field, and add some secret data. Response Wrapping

  9. Click Save.

  10. Click Copy secret > Wrap Secret. Response Wrapping

  11. Click the copy icon to copy the wrapping token value. Response Wrapping NOTE: You can click on the sensitive information toggle to show the wrapping token value.

  12. Sign out of the UI.

  13. Now, sign in with username bob and password training. Response Wrapping

    Notice that bob has no visibility to the secret/myapp/config path.

  14. Select Tools > Unwrap, and enter the wrapping token you copied. Response Wrapping

  15. Click Unwrap Data to reveal the secret. Response Wrapping

At this point, you can click Copy to save the data. Since the wrapping token is a single-use token that the user will not be able to unwrap the secrets again using the same token.

Additional Discussion: Cubbyhole

The cubbyhole secrets engine provides your own private secret storage space where no one else can read (including root). This comes handy when you want to store a password tied to your username that should not be shared with anyone.

The cubbyhole secrets engine is mounted at the cubbyhole/ prefix by default. The secrets you store in the cubbyhole/ path are tied to your token and all tokens are permitted to read and write to the cubbyhole secrets engine by the default policy.

$ vault policy read default

...
# Allow a token to manage its own cubbyhole
path "cubbyhole/*" {
    capabilities = ["create", "read", "update", "delete", "list"]
}
...

To test the cubbyhole secrets engine, perform the following steps. (NOTE: Keep using the apps token from Step 2 to ensure that you are logged in with non-root token.)

CLI command

Commands to write secrets to the cubbyhole secrets engine:

$ vault write cubbyhole/<PATH> <KEY>=<VALUE>

Commands to read secrets to the cubbyhole secrets engine:

$ vault read cubbyhole/<PATH>

Example:

Write some test secrets under cubbyhole/private/ path.

$ vault write cubbyhole/private/access-token token="123456789abcdefg87654321"
Success! Data written to: cubbyhole/private/access-token

Now, read the values from cubbyhole/private/access-token.

$ vault read cubbyhole/private/access-token
Key     Value
---     -----
token   123456789abcdefg87654321

Now, try to access the secret using the root token. It should NOT return the secret.

$ VAULT_TOKEN=<ROOT_TOKEN> vault read cubbyhole/private/access-token

No value found at cubbyhole/private/access-token

API call using cURL

The API to work with the cubbyhole secrets engine is very similar to secret secrets engine:

$ curl --header "X-Vault-Token: <TOKEN>" \
       --request POST \
       --data <SECRETS> \
       <VAULT_ADDRESS>/v1/cubbyhole/<PATH>

Example:

Write some test secrets under cubbyhole/private/ path.

$ curl --header "X-Vault-Token: e095129f-123a-4fef-c007-1f6a487cfa78" --request POST \
       --data '{"token": "123456789abcdefg87654321"}' \
       http://127.0.0.1:8200/v1/cubbyhole/private/access-token

Now, read the values from cubbyhole/private/access-token.

$ curl --header "X-Vault-Token: e095129f-123a-4fef-c007-1f6a487cfa78" \
       http://127.0.0.1:8200/v1/cubbyhole/private/access-token  | jq
{
 "request_id": "b2ff9f04-7a72-7eb0-672f-225b5eb652df",
 "lease_id": "",
 "renewable": false,
 "lease_duration": 0,
 "data": {
   "token": "123456789abcdefg87654321"
 },
 "wrap_info": null,
 "warnings": null,
 "auth": null
}

Try accessing the secret using the root token. It should NOT return the secret.

$ curl --header "X-Vault-Token: root" --request GET \
       http://127.0.0.1:8200/v1/cubbyhole/private/access-token  | jq
{
 "errors": []
}

Also, refer to Cubbyhole Secrets Engine (API).

Help and Reference