April 6 & 7
Learn about Vault, Consul, & more at HashiDays Sydney in Australia Register Now

Operations

[Enterprise] Performance Replication with Paths Filter

Paths filter is a new way of controlling which secrets are moved across clusters and physical regions as a result of replication. With replication filters, users can select which secrets engines will be replicated as part of a performance replication relationship.

By default, all non-local secrets engines and associated data are replicated as part of replication. The paths filter feature enables users to allow or deny which secrets engines are replicated, thereby allowing users to further control the movement of secrets across their infrastructure.

Performance Replication

Challenge

General Data Protection Regulation (GDPR) is designed to strengthen data protection and privacy for all individuals within the European Union. It requires that personally identifiable data not be physically transferred to locations outside the European Union unless the region or country has an equal rigor of data protection regulation as the EU.

Failure to abide by GDPR will result in fines as high as 20 million EUR or 4% of the global annual revenue (whichever greater).

Solution

Leverage Vault's paths filter feature to abide by data movements and sovereignty regulations while ensuring performance access across geographically distributed regions.

The Preparing for GDPR Compliance with HashiCorp Vault webinar discusses the GDPR compliance further in details.

Prerequisites

This intermediate Vault operations guide assumes that you have some working knowledge of Vault.

You need two Vault Enterprise clusters: one representing the EU cluster, and another representing the US cluster both backed by Consul for storage.

Scenario Introduction

An organization has a Vault cluster in EU and wish to span across the United States by setting up a secondary cluster and enable the performance replication. However, some data must remain in EU and should not be replicated to the US cluster.

Guide Scenario

Leverage the paths filter feature to deny the secrets from being replicated, that are subject to GDPR, from being replicated across the regions.

  1. Segment GDPR and non-GDPR secrets
  2. Enable performance replication with paths filter
  3. Secondary cluster re-authentication
  4. Verify the replication paths filter
  5. Enable a local secrets engine

Step 1: Segment GDPR and non-GDPR secrets

In the EU cluster (primary cluster), enable key/value secrets engines:

  • At EU_GDPR_data for GDPR data
  • At US_NON_GDPR_data for non-GDPR data localized for US

Also, create the following namespaces:

  • office_FR is used by the office branch in France
  • office_US is used by the office branch in US

CLI command / API call using cURL / Web UI

CLI command

  1. Enable the key/value v2 secrets engine at the EU_GDPR_data path.

    $ vault secrets enable -path=EU_GDPR_data kv-v2
    
  2. Enable the key/value v2 secrets engine at the US_NON_GDPR_data path.

    $ vault secrets enable -path=US_NON_GDPR_data kv-v2
    
  3. Create a namespace named office_FR.

    $ vault namespace create office_FR
    
  4. Create a namespace named office_US.

    $ vault namespace create office_US
    

API call using cURL

  1. Enable the key/value v2 secrets engine at the EU_GDPR_data path.

    $ curl --header "X-Vault-Token: <TOKEN>" \
          --request POST \
          --data '{"type":"kv-v2"}' \
          https://eu-west-1.compute.com:8200/v1/sys/mounts/EU_GDPR_data
    
  2. Enable the key/value v2 secrets engine at the US_NON_GDPR_data path.

    $ curl --header "X-Vault-Token: <TOKEN>" \
          --request POST \
          --data '{"type":"kv-v2"}' \
          https://eu-west-1.compute.com:8200/v1/sys/mounts/US_NON_GDPR_data
    
  3. Create a namespace named office_FR.

    $ curl --header "X-Vault-Token: <TOKEN>" \
          --request POST \
          http://127.0.0.1:8200/v1/sys/namespaces/office_FR
    
  4. Create a namespace named office_US.

    $ curl --header "X-Vault-Token: <TOKEN>" \
          --request POST \
          http://127.0.0.1:8200/v1/sys/namespaces/office_US
    

Web UI

  1. Open a web browser and launch the Vault UI (e.g. https://eu-west-1.compute.com:8200/ui) and then login.

  2. Select Enable new engine.

  3. Select KV from the list and then click Next.

  4. Enter EU_GDPR_data in the Path, and then click Enable Engine. GDPR KV

  5. Select Secrets and Enable new engine again.

  6. Select KV from the list and then click Next.

  7. Enter US_NON_GDPR_data in the Path, and then click Enable Engine. Non-GDPR KV

  8. Click Enable Engine to complete.

  9. Select Access.

  10. Select Namespaces and then click Create a namespace.

  11. Enter office_FR in the Path field.

  12. Click Save.

  13. Click Create a namespace.

  14. Enter office_US in the Path field, and then click Save. Web UI

Step 2: Enable performance replication with paths filters

CLI command / API call using cURL / Web UI

CLI command

  1. Enable performance replication on the primary cluster.

    $ vault write -f sys/replication/performance/primary/enable
    WARNING! The following warnings were returned from Vault:
    
    This cluster is being enabled as a primary for replication. Vault will be
    unavailable for a brief period and will resume service shortly.
    
  2. Generate a secondary token.

    $ vault write sys/replication/performance/primary/secondary-token id="secondary"
    Key                              Value
    ---                              -----
    wrapping_token:                  eyJhbGciOiJFUzUxMiIsInR5cCI6IkpXVCJ9.eyJhZGRyIjoiaHR0cDovLzE3Mi4zMS4yMC4xODA6ODIwMyIsImV4cCI6MTUyNTg0ODAxMywiaWF0IjoxNTI1ODQ2MjEzLCJqdGkiOiJlNTFiMjUxZi01ZTg2LTg4OWEtNGZmMy03NTQzMjRkNTdlMGQiLCJ0eXBlIjoid3JhcHBpbmcifQ.MIGGAkE2dDj3nmaoLHg7oldQ1iZPD0U8doyj3x3mQUVfTl8W99QYG8GM6VGVzhRPGvKctGriuo2oXN_8euWQb01M1y6n7gJBSu-qdXw-v2RieOyopAHls1bWhw4sO9Nlds8IDFA15vqkLXnq2g4_5lvlhxpP7B8dEOHvWXkHG4kJ_mKvrgR0dU0
    wrapping_accessor:               6ded4fb0-5e8c-2a37-1b3e-823673220348
    wrapping_token_ttl:              30m
    wrapping_token_creation_time:    2018-05-09 06:10:13.437421436 +0000 UTC
    wrapping_token_creation_path:    sys/replication/performance/primary/secondary-token
    
  3. Create a paths filter to deny EU_GDPR_data (kv-v2) and office_FR (namespace) from being replicated.

    $ vault write sys/replication/performance/primary/paths-filter/secondary  \
            mode="deny" paths="EU_GDPR_data/, office_FR"
    
  4. Enable performance replication on the secondary cluster by passing the wrapping_token obtained from the primary cluster.

    $ vault write sys/replication/performance/secondary/enable token="..."
    

API call using cURL

  1. Enable performance replication on the primary cluster.

    $ curl --header "X-Vault-Token: <TOKEN>" \
          --request POST \
          https://eu-west-1.compute.com:8200/v1/sys/replication/performance/primary/enable | jq
    {
      "request_id": "2d0a2729-430b-910c-e9ac-bf5161f73f13",
      "lease_id": "",
      "renewable": false,
      "lease_duration": 0,
      "data": null,
      "wrap_info": null,
      "warnings": [
        "This cluster is being enabled as a primary for replication. Vault will be unavailable for a brief period and will resume service shortly."
      ],
      "auth": null
    }
    
  2. Generate a secondary token.

    $ curl --header "X-Vault-Token: <TOKEN>" \
          --request POST \
          --data '{ "id": "secondary"}' \
          https://eu-west-1.compute.com:8200/v1/sys/replication/performance/primary/secondary-token | jq
      {
      "request_id": "",
      "lease_id": "",
      "renewable": false,
      "lease_duration": 0,
      "data": null,
      "wrap_info": {
        "token": "eyJhbGciOiJFUzUxMiIsInR5cCI6IkpXVCJ9.eyJhZGRyIjoiaHR0cDovLzEyNy4wLjAuMTo4MjAzIiwiZXhwIjoxNTI1ODI5Njc2LCJpYXQiOjE1MjU4Mjc4NzYsImp0aSI6IjAwNmVkMDdjLWQ0MzYtZWViYy01OWYwLTdiMTU0ZGFmMDNiMCIsInR5cGUiOiJ3cmFwcGluZyJ9.MIGHAkF6saWWL-oRQMJIoUnaUOHNkcoHZCBwQs6mSMjPBopMi8DkGCJGBrh4jgV2mSzwFY1r5Ne7O66HmuMpm40MsYqjAQJCANSco_Sx5q6FmQSfoY-HtsVO1_YKWF4O6B7gYCvPKYkODMIwe5orCSgmIDyXHZt-REPm0sfdk4ZNyRCIRK5hDWyQ",
        "accessor": "6ea2a4e2-2926-120f-f288-c2348c78fb3e",
        "ttl": 1800,
        "creation_time": "2018-05-09T01:04:36.514715311Z",
        "creation_path": "sys/replication/performance/primary/secondary-token"
      },
      "warnings": null,
      "auth": null
    }
    
  3. Create a paths filter to deny EU_GDPR_data (kv-v2) and office_FR (namespace) from being replicated.

    Create an API request payload containing the list of paths to deny.

    $ tee payload.json <<EOF
    {
      "mode": "deny",
      "paths": [ "EU_GDPR_data/", "office_FR/" ]
    }
    EOF
    

    Use the sys/replication/performance/primary/paths-filter/secondary endpoint to set the paths filter.

    $ curl --header "X-Vault-Token: <TOKEN>" \
          --request POST \
          --data @payload.json \
          https://eu-west-1.compute.com:8200/v1/sys/replication/performance/primary/paths-filter/secondary
    
  4. Enable performance replication on the secondary cluster.

    Create an API request payload containing the token obtained from the primary cluster.

    $ tee payload.json <<EOF
    {
      "token": "..."
    }
    EOF
    

    Use the sys/replication/performance/secondary/enable to enable the performance replication.

    $ curl --header "X-Vault-Token: <TOKEN>" \
          --request POST \
          --data @payload.json \
          https://us-central.compute.com:8201/v1/sys/replication/performance/secondary/enable
    

Web UI

  1. Select the arrow next to Status and click Enable under REPLICATION. Performance Replication - primary

  2. Select the Performance radio button. Performance Replication - primary

  3. Click Enable replication.

  4. Select the Add secondary. Performance Replication - primary

  5. Populate the Secondary ID field (e.g. secondary), and then select the Deny radio button under the Filtered paths.

  6. In the Paths in this deny field, select EU_GDPR_data. Performance Replication - primary

  7. Click Generate token. Performance Replication - primary

  8. Click Copy to copy the token.

  9. Now, launch the Vault UI for the secondary cluster (e.g. https://us-central.compute.com:8201/ui).

  10. Select the arrow next to Status and click Enable under REPLICATION.

  11. Check the Performance radio button, and then select secondary under the Cluster mode. Paste the token you copied from the primary. Performance Replication - secondary

  12. Click Enable Replication. Performance Replication - secondary

  13. Sign out of the secondary cluster UI.

Step 3: Secondary cluster re-authentication

$ vault operator unseal
Unseal Key (will be hidden): <primary_cluster_unseal_key>

The initial root token on the secondary no longer works; therefore, perform one of the following:

  • Option 1: Use the auth methods configured on the primary cluster to log into the secondary
  • Option 2: Generate a new root token using the primary's unseal key

Option 1

Enable and configure the userpass auth method on the primary cluster and create a new username and password.

$ vault auth enable userpass

Create a user with admin policy.

$ vault write auth/userpass/users/admin password="long-passw0rd" policy="admin"

Log into the secondary cluster using the enabled auth method.

$ vault login -method=userpass username=admin password="long-passw0rd"

Option 2

On the secondary cluster, generate a new root token using the primary cluster's unseal key.

First, generate and OTP.

$ vault operator generate-root -generate-otp
kqJzFJ5NqOWgjseqKnUZ2ZePCp

Initialize the root token generation using the OTP.

$ vault operator generate-root -init -otp=kqJzFJ5NqOWgjseqKnUZ2ZePCp
Nonce         35f246b1-0005-2702-05c2-37bdf7757a37
Started       true
Progress      0/1
Complete      false
OTP Length    26

Now, generate root. When prompted, enter the primary unseal key.

$ vault operator generate-root
Operation nonce: 35f246b1-0005-2702-05c2-37bdf7757a37
Unseal Key (will be hidden):

Nonce            35f246b1-0005-2702-05c2-37bdf7757a37
Started          true
Progress         1/1
Complete         true
Encoded Token    GF9+FTR5RToiF2QlAztQCA0GMhdmDFAFBEA

Finally, decode the encoded token using OTP.

$ vault operator generate-root \
        -decode=GF9+FTR5RToiF2QlAztQCA0GMhdmDFAFBEA \
        -otp=kqJzFJ5NqOWgjseqKnUZ2ZePCp

Log into the secondary cluster using the generated root token.

$ vault login <root_token>

Step 4: Verify the paths filter

Once the replication completes, verify that the secrets stored in the EU_GDPR_data never get replicated to the US cluster.

CLI command / API call using cURL / Web UI

CLI command

  1. On the EU (primary) cluster, write some secrets for testing.

    $ vault kv put EU_GDPR_data/secret pswd="password"
    Key              Value
    ---              -----
    created_time     2018-05-10T18:00:38.912587665Z
    deletion_time    n/a
    destroyed        false
    version          1
    

    Write some secret at US_NON_GDPR_data/secret.

    $ vault kv put US_NON_GDPR_data/secret apikey="my-api-key"
    Key              Value
    ---              -----
    created_time     2018-05-10T18:04:37.554665851Z
    deletion_time    n/a
    destroyed        false
    version          1
    
  2. List the existing namespaces.

    $ vault namespace list
    
    Keys
    ----
    office_FR/
    office_US/
    
  3. Now, from the US (secondary) cluster, read the secrets.

    Read the secrets at EU_GDPR_data/secret.

    $ vault kv get EU_GDPR_data/secret
    No value found at EU_GDPR_data/secret
    

    Read the secrets at US_NON_GDPR_data/secret.

    $ vault kv get US_NON_GDPR_data/secret
    ====== Metadata ======
    Key              Value
    ---              -----
    created_time     2018-05-10T18:09:07.717250408Z
    deletion_time    n/a
    destroyed        false
    version          1
    
    ===== Data =====
    Key       Value
    ---       -----
    apikey    my-api-key
    
  4. List the existing namespaces.

    $ vault namespace list
    
    Keys
    ----
    office_US/
    

    Notice that office_US is the only namespace listed.

API call using cURL

  1. On the EU (primary) cluster, write some secret.

    Create the API request payload containing some mock data.

    $ tee payload.json <<EOF
    {
      "data": {
        "pswd": "password"
      }
    }
    EOF
    

    Write some secret at EU_GDPR_data/secret.

    $ curl --header "X-Vault-Token: <TOKEN>" \
          --request POST \
          --data @payload.json \
          https://eu-west-1.compute.com:8200/v1/EU_GDPR_data/data/secret
    

    Create the API request payload containing some mock data.

    $ tee payload-us.json <<EOF
    {
    "data": {
      "apikey": "my-api-key"
    }
    }
    EOF
    

    Write some secret at US_NON_GDPR_data/secret.

    $ curl --header "X-Vault-Token: <TOKEN>" \
          --request POST \
          --data @payload-us.json \
          https://eu-west-1.compute.com:8200/v1/US_NON_GDPR_data/data/secret
    
  2. List the existing namespaces.

    $ curl --header "X-Vault-Token: <TOKEN>" \
          --request LIST
          http://127.0.0.1:8200/v1/education/sys/namespaces | jq -r ".data"
    {
      "key_info": {
        "office_FR/": {
          "id": "VszbY",
          "path": "office_FR/"
        },
        "office_US/": {
          "id": "L4WY2",
          "path": "office_US/"
        }
      },
      "office_US/": {
        "id": "L4WY2",
        "path": "office_US/"
      }
    },
    "keys": [
      "office_FR/",
      "office_US/"
    ]
    }
    
  3. Now, from the US (secondary) cluster, read the secrets.

    Read the secrets at EU_GDPR_data/secret.

    $ curl --header "X-Vault-Token: <TOKEN>" \
          https://us-central.compute.com:8201/v1/EU_GDPR_data/data/secret | jq
    {
      "errors": []
    }
    

    Read the secrets at US_NON_GDPR_data/secret.

    $ curl --header "X-Vault-Token: <TOKEN>" \
          https://us-central.compute.com:8201/v1/US_NON_GDPR_data/data/secret | jq
    {
      "request_id": "f5eedb5b-9406-c519-f5a5-070336c10205",
      "lease_id": "",
      "renewable": false,
      "lease_duration": 0,
      "data": null,
      "wrap_info": null,
      "warnings": [
        "Invalid path for a versioned K/V secrets engine. See the API docs for the appropriate API endpoints to use. If using the Vault CLI, use 'vault kv get' for this operation."
      ],
      "auth": null
    }
    
  4. List the existing namespaces.

    $ curl --header "X-Vault-Token: <TOKEN>" \
          --request LIST
          http://127.0.0.1:8200/v1/education/sys/namespaces | jq -r ".data"
    {
      "key_info": {
        "office_US/": {
          "id": "L4WY2",
          "path": "office_US/"
        }
      },
      "keys": [
        "office_US/"
      ]
    }
    

Notice that office_US is the only namespace listed.

Web UI

  1. On the EU (primary) cluster, select EU_GDPR_data > Create secret.

  2. Enter some test data and click Save.

    Secrets

  3. Repeat the step to write some secrets at the US_NON_GDPR_data path as well.

  4. Select Access.

  5. Select Namespaces. Both office_FR and office_US namespaces are listed.

  6. Return to the US (secondary) cluster, select US_NON_GDPR_data. You should be able to see the apikey under US_NON_GDPR_data/secret.

    Secrets

  7. The EU_GDPR_data data is not replicated, so you won't be able to see the secrets.

  8. Select Access.

  9. Select Namespaces and notice that office_US is the only namespace listed.

Step 5: Enable a local secrets engine

When replication is enabled, you can mark the secrets engine local only. Local secrets engines are not replicated or removed by replication.

Login to the secondary cluster and enable key/value secrets engine at US_ONLY_data to store secrets only valid for the US region.

CLI command / API call using cURL / Web UI

CLI command

Pass the -local flag.

$ vault secrets enable -local -path=US_ONLY_data kv-v2

API call using cURL

Create an API request payload containing the secrets engine type. Be sure to set the local parameter set to true.

$ tee payload.json <<EOF
{
  "type": "kv-v2",
  "local": true
}
EOF

Enable the secrets engine at US_ONLY_data.

$ curl --header "X-Vault-Token: <TOKEN>" \
       --request POST \
       --data @payload.json \
       https://us-central.compute.com:8201/v1/sys/mounts/US_ONLY_data

Web UI

Be sure to select the check box for Local to keep it mounted locally within the cluster.

Local Secret

Help and Reference