Performance Replication with Paths Filter

Enterprise Only: Performance Replication requires Vault Enterprise Premium license. To explore Vault Enterprise features, you can sign up for a free 30-day trial from here.

Paths filter is a new way of controlling which secrets are moved across clusters and physical regions as a result of replication. With replication filters, users can select which secrets engines will be replicated as part of a Performance Replication relationship.

By default, all non-local secrets engines and associated data are replicated as part of replication. The paths filter feature enables users to allow or deny which secrets engines are replicated, thereby allowing users to further control the movement of secrets across their infrastructure.

»Challenge

General Data Protection Regulation (GDPR) is designed to strengthen data protection and privacy for all individuals within the European Union. It requires that personally identifiable data not be physically transferred to locations outside the European Union unless the region or country has an equal rigor of data protection regulation as the EU.

Failure to abide by GDPR will result in fines as high as 20 million EUR or 4% of the global annual revenue (whichever greater).

»Solution

Leverage Vault's paths filter feature to abide by data movements and sovereignty regulations while ensuring performance access across geographically distributed regions. You can set filters based on the mount path of the secrets engines as well as namespaces.

»Prerequisites

This intermediate Vault operations tutorial assumes that you have some working knowledge of Vault.

You need two Vault Enterprise clusters: one representing the EU cluster, and another representing the US cluster both backed by Consul for storage.

NOTE: Refer to the Vault High Availability with Consul tutorial for configuring your Vault server.

»Scenario Introduction

An organization has a Vault cluster in EU and wish to span across the United States by setting up a secondary cluster and enable the Performance Replication. However, some data must remain in EU and should not be replicated to the US cluster

Leverage the paths filter feature to deny the secrets from being replicated, that are subject to GDPR, from being replicated across the regions.

Segment GDPR and non-GDPR secrets
Enable Performance Replication with paths filter
Secondary cluster re-authentication
Verify the replication paths filter
Enable a local secrets engine
Enable a local auth method

NOTE: Ensure that GDPR data is segmented by secret mount and deny the movement of those secret mounts to non-GDPR territories.

»Step 1: Segment GDPR and non-GDPR secrets

In the EU cluster (primary cluster), enable key/value secrets engines:

At EU_GDPR_data for GDPR data
At US_NON_GDPR_data for non-GDPR data localized for US

Also, create the following namespaces:

office_FR is used by the office branch in France
office_US is used by the office branch in US

Enable the key/value v2 secrets engine at the EU_GDPR_data path.

$ vault secrets enable -path=EU_GDPR_data kv-v2

$ vault secrets enable -path=EU_GDPR_data kv-v2

Enable the key/value v2 secrets engine at the US_NON_GDPR_data path.

$ vault secrets enable -path=US_NON_GDPR_data kv-v2

$ vault secrets enable -path=US_NON_GDPR_data kv-v2

Create a namespace named office_FR.

$ vault namespace create office_FR

$ vault namespace create office_FR

Create a namespace named office_US.

$ vault namespace create office_US

$ vault namespace create office_US

»Step 2: Enable Performance Replication with paths filters

Enable Performance Replication on the primary cluster.

$ vault write -f sys/replication/performance/primary/enable
WARNING! The following warnings were returned from Vault:

This cluster is being enabled as a primary for replication. Vault will be
unavailable for a brief period and will resume service shortly.

$ vault write -f sys/replication/performance/primary/enable
WARNING! The following warnings were returned from Vault:

This cluster is being enabled as a primary for replication. Vault will be
unavailable for a brief period and will resume service shortly.

NOTE: If the primary's cluster address is not directly accessible and must be accessed via an alternate path/address (e.g. through a TCP-based load balancer), use the primary_cluster_addr parameter to specify the address to be used by the secondaries. Otherwise, the secondaries use the configured cluster address to connect to the primary. See the Vault High Availability with Consul tutorial for an example Vault server configuration.

Generate a secondary token.

$ vault write sys/replication/performance/primary/secondary-token id="secondary"

 Key                              Value
 ---                              -----
 wrapping_token:                  eyJhbGciOiJFUzUxMiIsInR5cCI6IkpXVCJ9.eyJhY2Nlc3NvciI6IiIsImFkZHIiOiJodHRwOi8vMTI3LjAuMC4xOjgyMDAiLCJleHAiOjE2NTI0NzMzNzUsImlhdCI6MTY1MjQ3MTU3NSwianRpIjoiaHZzLnVJTElBUE5tY0xKbkpQd1oyUEVHSkdkcCIsIm5iZiI6MTY1MjQ3MTU3MCwidHlwZSI6IndyYXBwaW5nIn0.AeELVB1Ji-4pF-nWE03-Pj3YLkCt5Zi90_knmkG5nZUhPFdmZR0SalqeW5U9CA0HPitPJXnIsdLfulo_SzLmbZr6Ad_r31TazM3AiDzS1EL6vwSkhL9vNc323pWSPiqCXTZkGPffgQ11n-1JnluqjKlx-S7-cC7pA4PL0hZB0eguaMKQ
 wrapping_accessor:               YBUiHKTkfnlG9WXsYzsdCBlu
 wrapping_token_ttl:              30m
 wrapping_token_creation_time:    2022-05-13 12:52:55.809933 -0700 PDT
 wrapping_token_creation_path:    sys/replication/performance/primary/secondary-token

$ vault write sys/replication/performance/primary/secondary-token id="secondary"

 Key                              Value
 ---                              -----
 wrapping_token:                  eyJhbGciOiJFUzUxMiIsInR5cCI6IkpXVCJ9.eyJhY2Nlc3NvciI6IiIsImFkZHIiOiJodHRwOi8vMTI3LjAuMC4xOjgyMDAiLCJleHAiOjE2NTI0NzMzNzUsImlhdCI6MTY1MjQ3MTU3NSwianRpIjoiaHZzLnVJTElBUE5tY0xKbkpQd1oyUEVHSkdkcCIsIm5iZiI6MTY1MjQ3MTU3MCwidHlwZSI6IndyYXBwaW5nIn0.AeELVB1Ji-4pF-nWE03-Pj3YLkCt5Zi90_knmkG5nZUhPFdmZR0SalqeW5U9CA0HPitPJXnIsdLfulo_SzLmbZr6Ad_r31TazM3AiDzS1EL6vwSkhL9vNc323pWSPiqCXTZkGPffgQ11n-1JnluqjKlx-S7-cC7pA4PL0hZB0eguaMKQ
 wrapping_accessor:               YBUiHKTkfnlG9WXsYzsdCBlu
 wrapping_token_ttl:              30m
 wrapping_token_creation_time:    2022-05-13 12:52:55.809933 -0700 PDT
 wrapping_token_creation_path:    sys/replication/performance/primary/secondary-token

Create a paths filter to deny EU_GDPR_data (kv-v2) and office_FR (namespace) from being replicated.

$ vault write sys/replication/performance/primary/paths-filter/secondary  \
       mode="deny" paths="EU_GDPR_data/, office_FR"

$ vault write sys/replication/performance/primary/paths-filter/secondary  \
       mode="deny" paths="EU_GDPR_data/, office_FR"

Enable Performance Replication on the secondary cluster by passing the wrapping_token obtained from the primary cluster.
```
$ vault write sys/replication/performance/secondary/enable token="..."
```
```
$ vault write sys/replication/performance/secondary/enable token="..."
```
NOTE: This will immediately clear all data in the secondary cluster.

»Step 3: Secondary cluster re-authentication

NOTE: From this point and on, the secondary cluster requires the primary cluster's unseal key to unseal. If the secondary is in an HA cluster, each standby node needs the primary cluster's unseal keys to unseal. The secondary cluster mirrors the configuration of its primary cluster's backends such as auth methods, secrets engines, audit devices, etc. It uses the primary as the source of truth and passes token requests to the primary.

$ vault operator unseal
Unseal Key (will be hidden): <primary_cluster_unseal_key>

$ vault operator unseal
Unseal Key (will be hidden): <primary_cluster_unseal_key>

The initial root token on the secondary no longer works; therefore, perform one of the following:

Option 1: Use the auth methods configured on the primary cluster to log into the secondary
Option 2: Generate a new root token using the primary's unseal key
Option 3: Generate a batch token to use across replication clusters

»Option 1

On the primary cluster, create a superuser policy.

$ vault policy write superuser -<<EOF
path "*" {
  capabilities = ["create", "read", "update", "delete", "list", "sudo"]
}
EOF

$ vault policy write superuser -<<EOF
path "*" {
  capabilities = ["create", "read", "update", "delete", "list", "sudo"]
}
EOF

Enable userpass auth method.

$ vault auth enable userpass

$ vault auth enable userpass

Create a new user, tester in userpass where the password is changeme and superuser policy is attached.

$ vault write auth/userpass/users/tester password="changeme" policies="superuser"

$ vault write auth/userpass/users/tester password="changeme" policies="superuser"

NOTE: If you are not familiar with policies, refer to the policies tutorial.

Log into the secondary cluster using the enabled auth method.

$ vault login -method=userpass username=tester password="changeme"

Success! You are now authenticated. The token information displayed below
is already stored in the token helper. You do NOT need to run "vault login"
again. Future Vault requests will automatically use this token.

Key                    Value
---                    -----
token                  hvs.CAESIMxjDq_LN3vJ6IhvV4N6d-xi4xqVzU1KCrkBaeMmreYMGiEKHGh2cy5wWENSSDBHOGVPU08zRTRCMW1MRVBMMTkQwQQ
token_accessor         2AB9ksQ80lB7bWeQtMIlbqgM
token_duration         768h
token_renewable        true
token_policies         ["default" "superuser"]
identity_policies      []
policies               ["default" "superuser"]
token_meta_username    tester

$ vault login -method=userpass username=tester password="changeme"

Success! You are now authenticated. The token information displayed below
is already stored in the token helper. You do NOT need to run "vault login"
again. Future Vault requests will automatically use this token.

Key                    Value
---                    -----
token                  hvs.CAESIMxjDq_LN3vJ6IhvV4N6d-xi4xqVzU1KCrkBaeMmreYMGiEKHGh2cy5wWENSSDBHOGVPU08zRTRCMW1MRVBMMTkQwQQ
token_accessor         2AB9ksQ80lB7bWeQtMIlbqgM
token_duration         768h
token_renewable        true
token_policies         ["default" "superuser"]
identity_policies      []
policies               ["default" "superuser"]
token_meta_username    tester

»Option 2

On the secondary cluster, generate a new root token using the primary cluster's unseal key.

On the secondary cluster, initialize the root generation operation.

$ vault operator generate-root -init

$ vault operator generate-root -init

Example output:

A One-Time-Password has been generated for you and is shown in the OTP field.
You will need this value to decode the resulting root token, so keep it safe.
Nonce         096a0930-b798-a4e4-371b-6027c8cbb09b
Started       true
Progress      0/3
Complete      false
OTP           Ibrifu9LvkEpYHi02TL8PMhegR
OTP Length    26

A One-Time-Password has been generated for you and is shown in the OTP field.
You will need this value to decode the resulting root token, so keep it safe.
Nonce         096a0930-b798-a4e4-371b-6027c8cbb09b
Started       true
Progress      0/3
Complete      false
OTP           Ibrifu9LvkEpYHi02TL8PMhegR
OTP Length    26

Nonce and one-time password (OTP) are generated. The nonce value should be distributed to all unseal key (recovery key if auto-unseal is used) holders. You will need the OTP to decode the generated root token.

Execute the generate-root command until the threshold is reached.

$ vault operator generate-root

Operation nonce: 096a0930-b798-a4e4-371b-6027c8cbb09b
Unseal Key (will be hidden):

$ vault operator generate-root

Operation nonce: 096a0930-b798-a4e4-371b-6027c8cbb09b
Unseal Key (will be hidden):

When prompted, enter the unseal key.

Nonce       096a0930-b798-a4e4-371b-6027c8cbb09b
Started     true
Progress    1/3
Complete    false

Nonce       096a0930-b798-a4e4-371b-6027c8cbb09b
Started     true
Progress    1/3
Complete    false

When the threshold of unseal keys (or recovery keys) are supplied, the output includes the encoded root token.

$ vault operator generate-root

$ vault operator generate-root

Example output:

Operation nonce: 096a0930-b798-a4e4-371b-6027c8cbb09b
Unseal Key (will be hidden):

Nonce            096a0930-b798-a4e4-371b-6027c8cbb09b
Started          true
Progress         3/3
Complete         true
Encoded Token    Okw3ESAkDS4dHg4gOh4PRFNgGFMjBQ0NDTA

Operation nonce: 096a0930-b798-a4e4-371b-6027c8cbb09b
Unseal Key (will be hidden):

Nonce            096a0930-b798-a4e4-371b-6027c8cbb09b
Started          true
Progress         3/3
Complete         true
Encoded Token    Okw3ESAkDS4dHg4gOh4PRFNgGFMjBQ0NDTA

Finally, decode the encoded token using OTP.

$ vault operator generate-root \
    -decode=<encoded_token> \
    -otp=<otp>

$ vault operator generate-root \
    -decode=<encoded_token> \
    -otp=<otp>

Example:

$ vault operator generate-root \
    -decode="Okw3ESAkDS4dHg4gOh4PRFNgGFMjBQ0NDTA" \
    -otp="Ibrifu9LvkEpYHi02TL8PMhegR"

hvs.Av0hzOyHCBL55YbN7HTuSz9J

$ vault operator generate-root \
    -decode="Okw3ESAkDS4dHg4gOh4PRFNgGFMjBQ0NDTA" \
    -otp="Ibrifu9LvkEpYHi02TL8PMhegR"

hvs.Av0hzOyHCBL55YbN7HTuSz9J

Log into the secondary cluster using the generated root token.

$ vault login <root_token>

$ vault login <root_token>

Example:

$ vault login hvs.ZVwepu6BAPErDVqgHmMETDxW

Success! You are now authenticated. The token information displayed below
is already stored in the token helper. You do NOT need to run "vault login"
again. Future Vault requests will automatically use this token.

Key                  Value
---                  -----
token                hvs.ZVwepu6BAPErDVqgHmMETDxW
token_accessor       7IBRVnNnBPu54ryGpTwq4mHX
token_duration       ∞
token_renewable      false
token_policies       ["root"]
identity_policies    []
policies             ["root"]

$ vault login hvs.ZVwepu6BAPErDVqgHmMETDxW

Success! You are now authenticated. The token information displayed below
is already stored in the token helper. You do NOT need to run "vault login"
again. Future Vault requests will automatically use this token.

Key                  Value
---                  -----
token                hvs.ZVwepu6BAPErDVqgHmMETDxW
token_accessor       7IBRVnNnBPu54ryGpTwq4mHX
token_duration       ∞
token_renewable      false
token_policies       ["root"]
identity_policies    []
policies             ["root"]

»Option 3

Unlike service tokens, batch tokens can be used across the Performance Replication clusters. So, you can generate a batch token on the primary cluster and use it to log into the secondary cluster.

Similar to the option 1, create a policy to attach to the token on the primary cluster.

$ vault policy write superuser -<<EOF
path "*" {
  capabilities = ["create", "read", "update", "delete", "list", "sudo"]
}
EOF

$ vault policy write superuser -<<EOF
path "*" {
  capabilities = ["create", "read", "update", "delete", "list", "sudo"]
}
EOF

Generate an orphan batch token which is valid for 24 hours.

$ vault token create -type=batch -orphan -ttl=24h -policy=superuser

$ vault token create -type=batch -orphan -ttl=24h -policy=superuser

Important: The batch token must be an orphan token (-orphan) since the secondary cluster will not be able to ensure the validity of its parent token.

Example output:

Key                  Value
---                  -----
token                hvb.AAAAAQKey4-YfVFBjOAv6m8Rccz8NGIeyS5lMHjU44-GuqxhiMi5vqq2D5X0wgwXXGa1ASbnSzuRurCMMjIexruOn5bt7XjFtykBjNOyppvvRFY_jtX3YlrSJ9kpFsxagFXnvmPVtg
token_accessor       n/a
token_duration       24h
token_renewable      false
token_policies       ["default" "superuser"]
identity_policies    []
policies             ["default" "superuser"]

Key                  Value
---                  -----
token                hvb.AAAAAQKey4-YfVFBjOAv6m8Rccz8NGIeyS5lMHjU44-GuqxhiMi5vqq2D5X0wgwXXGa1ASbnSzuRurCMMjIexruOn5bt7XjFtykBjNOyppvvRFY_jtX3YlrSJ9kpFsxagFXnvmPVtg
token_accessor       n/a
token_duration       24h
token_renewable      false
token_policies       ["default" "superuser"]
identity_policies    []
policies             ["default" "superuser"]

Log into the secondary cluster using the generated batch token.

$ vault login <batch_token>

$ vault login <batch_token>

Example:

$ vault login hvb.AAAAAQKey4-YfVFBjOAv6m8Rccz8NGIeyS5lMHjU44-GuqxhiMi5vqq2D5X0wgwXXGa1ASbnSzuRurCMMjIexruOn5bt7XjFtykBjNOyppvvRFY_jtX3YlrSJ9kpFsxagFXnvmPVtg

Success! You are now authenticated. The token information displayed below
is already stored in the token helper. You do NOT need to run "vault login"
again. Future Vault requests will automatically use this token.

Key                  Value
---                  -----
token                hvb.AAAAAQKey4-YfVFBjOAv6m8Rccz8NGIeyS5lMHjU44-GuqxhiMi5vqq2D5X0wgwXXGa1ASbnSzuRurCMMjIexruOn5bt7XjFtykBjNOyppvvRFY_jtX3YlrSJ9kpFsxagFXnvmPVtg
token_accessor       n/a
token_duration       23h57m55s
token_renewable      false
token_policies       ["default" "superuser"]
identity_policies    []
policies             ["default" "superuser"]

$ vault login hvb.AAAAAQKey4-YfVFBjOAv6m8Rccz8NGIeyS5lMHjU44-GuqxhiMi5vqq2D5X0wgwXXGa1ASbnSzuRurCMMjIexruOn5bt7XjFtykBjNOyppvvRFY_jtX3YlrSJ9kpFsxagFXnvmPVtg

Success! You are now authenticated. The token information displayed below
is already stored in the token helper. You do NOT need to run "vault login"
again. Future Vault requests will automatically use this token.

Key                  Value
---                  -----
token                hvb.AAAAAQKey4-YfVFBjOAv6m8Rccz8NGIeyS5lMHjU44-GuqxhiMi5vqq2D5X0wgwXXGa1ASbnSzuRurCMMjIexruOn5bt7XjFtykBjNOyppvvRFY_jtX3YlrSJ9kpFsxagFXnvmPVtg
token_accessor       n/a
token_duration       23h57m55s
token_renewable      false
token_policies       ["default" "superuser"]
identity_policies    []
policies             ["default" "superuser"]

»Step 4: Verify the paths filter

Once the replication completes, verify that the secrets stored in the EU_GDPR_data never get replicated to the US cluster.

On the EU (primary) cluster, write some secrets for testing.

$ vault kv put EU_GDPR_data/secret pswd="password"

 ====== Secret Path ======
 EU_GDPR_data/data/secret

 ======= Metadata =======
 Key                Value
 ---                -----
 created_time       2022-05-13T19:54:14.100461Z
 custom_metadata    <nil>
 deletion_time      n/a
 destroyed          false
 version            1

$ vault kv put EU_GDPR_data/secret pswd="password"

 ====== Secret Path ======
 EU_GDPR_data/data/secret

 ======= Metadata =======
 Key                Value
 ---                -----
 created_time       2022-05-13T19:54:14.100461Z
 custom_metadata    <nil>
 deletion_time      n/a
 destroyed          false
 version            1

Write some secret at US_NON_GDPR_data/secret.

$ vault kv put US_NON_GDPR_data/secret apikey="my-api-key"

 ======== Secret Path ========
 US_NON_GDPR_data/data/secret

 ======= Metadata =======
 Key                Value
 ---                -----
 created_time       2022-05-13T19:55:05.044739Z
 custom_metadata    <nil>
 deletion_time      n/a
 destroyed          false
 version            1

$ vault kv put US_NON_GDPR_data/secret apikey="my-api-key"

 ======== Secret Path ========
 US_NON_GDPR_data/data/secret

 ======= Metadata =======
 Key                Value
 ---                -----
 created_time       2022-05-13T19:55:05.044739Z
 custom_metadata    <nil>
 deletion_time      n/a
 destroyed          false
 version            1

List the existing namespaces.

$ vault namespace list

Keys
----
office_FR/
office_US/

$ vault namespace list

Keys
----
office_FR/
office_US/

Now, from the US (secondary) cluster, read the secrets.

Read the secrets at EU_GDPR_data/secret.

$ vault kv get EU_GDPR_data/secret
No value found at EU_GDPR_data/secret

$ vault kv get EU_GDPR_data/secret
No value found at EU_GDPR_data/secret

Read the secrets at US_NON_GDPR_data/secret.

$ vault kv get US_NON_GDPR_data/secret

 ======== Secret Path ========
 US_NON_GDPR_data/data/secret

 ======= Metadata =======
 Key                Value
 ---                -----
 created_time       2022-05-13T19:55:05.044739Z
 custom_metadata    <nil>
 deletion_time      n/a
 destroyed          false
 version            1

 ===== Data =====
 Key       Value
 ---       -----
 apikey    my-api-key

$ vault kv get US_NON_GDPR_data/secret

 ======== Secret Path ========
 US_NON_GDPR_data/data/secret

 ======= Metadata =======
 Key                Value
 ---                -----
 created_time       2022-05-13T19:55:05.044739Z
 custom_metadata    <nil>
 deletion_time      n/a
 destroyed          false
 version            1

 ===== Data =====
 Key       Value
 ---       -----
 apikey    my-api-key

List the existing namespaces.

$ vault namespace list

Keys
----
office_US/

$ vault namespace list

Keys
----
office_US/

Notice that office_US is the only namespace listed.

NOTE: Refer to the Monitoring Vault Replication tutorial for replication health check.

»Step 5: Enable local secrets engine

When replication is enabled, you can mark a secrets engine or auth method local only. Local secrets engines are not replicated or removed by replication.

Enable the local secrets engine by logging into the secondary cluster and defining a key/value secrets engine at the path US_ONLY_data to store secrets valid only for the US region.

Pass the -local flag.

$ vault secrets enable -local -path=US_ONLY_data kv-v2

$ vault secrets enable -local -path=US_ONLY_data kv-v2

»Step 6: Enable local auth method

NOTE: In Vault 1.9, the computation of client metrics has changed so that tokens generated by the local auth methods count towards clients (unique entities) instead of non-entity tokens. This prevents improper billing due to the non-entity tokens issued by the local auth methods. Prior to Vault 1.9, local auth methods contributed towards non-entity token counts. To learn more about client count, see the usage metrics tutorial for more details. If you are unfamiliar with the entity concept, refer to the Identity: Entities and Groups tutorial.

Enable the local auth method by logging into the secondary cluster and defining a username and password auth method at the path US_ONLY_userpass to handle logins valid only for the US region.

Pass the -local flag.

$ vault auth enable -local -path=US_ONLY_userpass userpass

$ vault auth enable -local -path=US_ONLY_userpass userpass

SECURITY NOTE: If your goal in marking an auth method as local is to comply with GDPR guidelines, then you must take care to not set the data pertaining to local auth mount or local auth mount aliases in the metadata of the associated entity. As of Vault 1.9, metadata related to local auth mount aliases can be stored as custom_metadata on the alias itself which will also be retained locally to the cluster. The local auth methods documentation has more details.

»Help and Reference

Preparing for GDPR Compliance with HashiCorp Vault webinar
Preparing for GDPR Compliance with HashiCorp Vault blog post
Create Paths Filter (API)
Performance Replication and Disaster Recovery (DR) Replication
Monitoring Vault Replication tutorial

Infrastructure

Security

Applications

Networking

cloud

»Challenge

»Solution

»Prerequisites

»Scenario Introduction

»Step 1: Segment GDPR and non-GDPR secrets

»Step 2: Enable Performance Replication with paths filters

»Step 3: Secondary cluster re-authentication

»Option 1

»Option 2

»Option 3

»Step 4: Verify the paths filter

»Step 5: Enable local secrets engine

»Step 6: Enable local auth method

»Help and Reference