Enterprise Only: This feature requires Vault Enterprise with the Multi-Datacenter & Scale Module.
Paths filter is a new way of controlling which secrets are moved across clusters and physical regions as a result of replication. With replication filters, users can select which secrets engines will be replicated as part of a performance replication relationship.
By default, all non-local secrets engines and associated data are replicated as part of replication. The paths filter feature enables users to allow or deny which secrets engines are replicated, thereby allowing users to further control the movement of secrets across their infrastructure.
»Challenge
General Data Protection Regulation (GDPR) is designed to strengthen data protection and privacy for all individuals within the European Union. It requires that personally identifiable data not be physically transferred to locations outside the European Union unless the region or country has an equal rigor of data protection regulation as the EU.
Failure to abide by GDPR will result in fines as high as 20 million EUR or 4% of the global annual revenue (whichever greater).
»Solution
Leverage Vault's paths filter feature to abide by data movements and sovereignty regulations while ensuring performance access across geographically distributed regions. You can set filters based on the mount path of the secrets engines as well as namespaces.
»Prerequisites
This intermediate Vault operations tutorial assumes that you have some working knowledge of Vault.
You need two Vault Enterprise clusters: one representing the EU cluster, and another representing the US cluster both backed by Consul for storage.
NOTE: Refer to the Vault High Availability with Consul tutorial for configuring your Vault server.
»Scenario Introduction
An organization has a Vault cluster in EU and wish to span across the United States by setting up a secondary cluster and enable the performance replication. However, some data must remain in EU and should not be replicated to the US cluster.
Leverage the paths filter feature to deny the secrets from being replicated, that are subject to GDPR, from being replicated across the regions.
- Segment GDPR and non-GDPR secrets
- Enable performance replication with paths filter
- Secondary cluster re-authentication
- Verify the replication paths filter
- Enable a local secrets engine
NOTE: Ensure that GDPR data is segmented by secret mount and deny the movement of those secret mounts to non-GDPR territories.
»Step 1: Segment GDPR and non-GDPR secrets
In the EU cluster (primary cluster), enable key/value secrets engines:
- At
EU_GDPR_data
for GDPR data - At
US_NON_GDPR_data
for non-GDPR data localized for US
Also, create the following namespaces:
Enable the key/value v2 secrets engine at the
EU_GDPR_data
path.$ vault secrets enable -path=EU_GDPR_data kv-v2
Enable the key/value v2 secrets engine at the
US_NON_GDPR_data
path.$ vault secrets enable -path=US_NON_GDPR_data kv-v2
Create a namespace named
office_FR
.$ vault namespace create office_FR
Create a namespace named
office_US
.$ vault namespace create office_US
»Step 2: Enable performance replication with paths filters
Enable performance replication on the primary cluster.
$ vault write -f sys/replication/performance/primary/enable WARNING! The following warnings were returned from Vault: This cluster is being enabled as a primary for replication. Vault will be unavailable for a brief period and will resume service shortly.
NOTE: If the primary's cluster address is not directly accessible and must be accessed via an alternate path/address (e.g. through a TCP-based load balancer), use the
primary_cluster_addr
parameter to specify the address to be used by the secondaries. Otherwise, the secondaries use the configured cluster address to connect to the primary. See the Vault High Availability with Consul tutorial for an example Vault server configuration.Generate a secondary token.
$ vault write sys/replication/performance/primary/secondary-token id="secondary" Key Value --- ----- wrapping_token: eyJhbGciOiJFUzUxMiIsInR5cCI6IkpXVCJ9.eyJhZGRyIjoiaHR0cDovLzE3Mi4zMS4yMC4xODA6ODIwMyIsImV4cCI6MTUyNTg0ODAxMywiaWF0IjoxNTI1ODQ2MjEzLCJqdGkiOiJlNTFiMjUxZi01ZTg2LTg4OWEtNGZmMy03NTQzMjRkNTdlMGQiLCJ0eXBlIjoid3JhcHBpbmcifQ.MIGGAkE2dDj3nmaoLHg7oldQ1iZPD0U8doyj3x3mQUVfTl8W99QYG8GM6VGVzhRPGvKctGriuo2oXN_8euWQb01M1y6n7gJBSu-qdXw-v2RieOyopAHls1bWhw4sO9Nlds8IDFA15vqkLXnq2g4_5lvlhxpP7B8dEOHvWXkHG4kJ_mKvrgR0dU0 wrapping_accessor: 6ded4fb0-5e8c-2a37-1b3e-823673220348 wrapping_token_ttl: 30m wrapping_token_creation_time: 2018-05-09 06:10:13.437421436 +0000 UTC wrapping_token_creation_path: sys/replication/performance/primary/secondary-token
Create a paths filter to deny
EU_GDPR_data
(kv-v2) andoffice_FR
(namespace) from being replicated.$ vault write sys/replication/performance/primary/paths-filter/secondary \ mode="deny" paths="EU_GDPR_data/, office_FR"
Enable performance replication on the secondary cluster by passing the
wrapping_token
obtained from the primary cluster.$ vault write sys/replication/performance/secondary/enable token="..."
NOTE: This will immediately clear all data in the secondary cluster.
»Step 3: Secondary cluster re-authentication
NOTE: From this point and on, the secondary cluster requires the primary cluster's unseal key to unseal. If the secondary is in an HA cluster, each standby node needs the primary cluster's unseal keys to unseal. The secondary cluster mirrors the configuration of its primary cluster's backends such as auth methods, secrets engines, audit devices, etc. It uses the primary as the source of truth and passes token requests to the primary.
$ vault operator unseal
Unseal Key (will be hidden): <primary_cluster_unseal_key>
The initial root token on the secondary no longer works; therefore, perform one of the following:
- Option 1: Use the auth methods configured on the primary cluster to log into the secondary
- Option 2: Generate a new root token using the primary's unseal key
»Option 1
On the primary cluster, create a superuser
policy.
$ vault policy write superuser -<<EOF
path "*" {
capabilities = ["create", "read", "update", "delete", "list", "sudo"]
}
EOF
Enable userpass
auth method.
$ vault auth enable userpass
Create a new user, tester
in userpass where the password is changeme
and
superuser
policy is attached.
$ vault write auth/userpass/users/tester password="changeme" policies="superuser"
NOTE: If you are not familiar with policies, refer to the policies tutorial.
Log into the secondary cluster using the enabled auth method.
$ vault login -method=userpass username=tester password="changeme"
Key Value
--- -----
token s.ZQ5ZpcrganMNxwp00CZymWC2
token_accessor XvDd5guLhZB88BfuWfsNKfMr
token_duration 768h
token_renewable true
token_policies ["default" "superuser"]
identity_policies []
policies ["default" "superuser"]
token_meta_username tester
»Option 2
On the secondary cluster, generate a new root token using the primary cluster's unseal key.
First, initialize the root generation operation.
$ vault operator generate-root -init
A One-Time-Password has been generated for you and is shown in the OTP field.
You will need this value to decode the resulting root token, so keep it safe.
Nonce 096a0930-b798-a4e4-371b-6027c8cbb09b
Started true
Progress 0/3
Complete false
OTP Ibrifu9LvkEpYHi02TL8PMhegR
OTP Length 26
Nonce and one-time password (OTP) are generated. The nonce value should be distributed to all unseal key (recovery key if auto-unseal is used) holders.
$ vault operator generate-root
Operation nonce: 096a0930-b798-a4e4-371b-6027c8cbb09b
Unseal Key (will be hidden):
When prompted, enter the unseal key.
Nonce 096a0930-b798-a4e4-371b-6027c8cbb09b
Started true
Progress 1/3
Complete false
When the threshold of unseal keys (or recovery keys) are supplied, the final user will also get the encoded root token.
$ vault operator generate-root
Operation nonce: 096a0930-b798-a4e4-371b-6027c8cbb09b
Unseal Key (will be hidden):
Nonce 096a0930-b798-a4e4-371b-6027c8cbb09b
Started true
Progress 3/3
Complete true
Encoded Token Okw3ESAkDS4dHg4gOh4PRFNgGFMjBQ0NDTA
Finally, decode the encoded token using OTP.
$ vault operator generate-root \
-decode="Okw3ESAkDS4dHg4gOh4PRFNgGFMjBQ0NDTA" \
-otp="Ibrifu9LvkEpYHi02TL8PMhegR"
s.ExFQ4bkuKPcVfta4TksHehjb
Log into the secondary cluster using the generated root token.
$ vault login <root_token>
»Step 4: Verify the paths filter
Once the replication completes, verify that the secrets stored in the
EU_GDPR_data
never get replicated to the US cluster.
On the EU (primary) cluster, write some secrets for testing.
$ vault kv put EU_GDPR_data/secret pswd="password" Key Value --- ----- created_time 2018-05-10T18:00:38.912587665Z deletion_time n/a destroyed false version 1
Write some secret at
US_NON_GDPR_data/secret
.$ vault kv put US_NON_GDPR_data/secret apikey="my-api-key" Key Value --- ----- created_time 2018-05-10T18:04:37.554665851Z deletion_time n/a destroyed false version 1
List the existing namespaces.
$ vault namespace list Keys ---- office_FR/ office_US/
Now, from the US (secondary) cluster, read the secrets.
Read the secrets at
EU_GDPR_data/secret
.$ vault kv get EU_GDPR_data/secret No value found at EU_GDPR_data/secret
Read the secrets at
US_NON_GDPR_data/secret
.$ vault kv get US_NON_GDPR_data/secret ====== Metadata ====== Key Value --- ----- created_time 2018-05-10T18:09:07.717250408Z deletion_time n/a destroyed false version 1 ===== Data ===== Key Value --- ----- apikey my-api-key
List the existing namespaces.
$ vault namespace list Keys ---- office_US/
Notice that
office_US
is the only namespace listed.
NOTE: Refer to the Monitoring Vault Replication tutorial for replication health check.
»Step 5: Enable a local secrets engine
When replication is enabled, you can mark the secrets engine local only. Local secrets engines are not replicated or removed by replication.
Login to the secondary cluster and enable key/value secrets engine at
US_ONLY_data
to store secrets only valid for the US region.
Pass the -local
flag.
$ vault secrets enable -local -path=US_ONLY_data kv-v2
»Help and Reference
- Preparing for GDPR Compliance with HashiCorp Vault webinar
- Preparing for GDPR Compliance with HashiCorp Vault blog post
- Create Paths Filter (API)
- Performance Replication and Disaster Recovery (DR) Replication
- Monitoring Vault Replication tutorial