[Tech Preview] Tokenize Data with Transform Secrets Engine

NOTE: This tutorial introduces tokenization transformation which is currently available as Technical Preview.

»Challenge

When encrypting sensitive data, preservation of the original data format or length may be required to meet certain industry standards such as HIPAA or PCI. To fulfill this requirement, the transform secrets engine performs format preserving encryption (FPE).

However, there are organizations that care more about the irreversibility of the tokenized data and not so much about preserving the original data format. Therefore, the transform secrets engine's FPE transformation may not meet the governance, risk and compliance (GRC) strategy they are looking for due to the use of reversible cryptography to perform FPE.

»Solution

In Vault 1.6, transform secrets engine introduced a new data transformation method to tokenize sensitive data stored outside of Vault. Tokenization replaces sensitive data with unique values (tokens) that are unrelated to the original value in any algorithmic sense. Therefore, those tokens cannot risk exposing the plaintext satisfying the PCI-DSS guidance.

Tokenization

»Characteristics of the tokenization transformation:

Non-reversible identification: Protect data pursuant to requirements for data irreversibility (PCI-DSS, GDPR, etc.)
Integrated Metadata: Supports metadata for identifying data type and purpose
Extreme scale and performance: Support for performantly managing billions of tokens across clouds as well as on-premise

»Prerequisites

To perform the tasks described in this tutorial, you need to have a Vault Enterprise v1.6 or later with Advanced Data Protection module.

NOTE: To explore Vault Enterprise features, you can sign up for a free 30-day trial.

An interactive tutorial is also available if you do not have a Vault environment to perform the steps described in this tutorial. Click the Show Terminal button to start.

»Policy requirements

NOTE: For the purpose of this tutorial, you can use root token to work with Vault. However, it is recommended that root tokens are only used for just enough initial setup or in emergencies. As a best practice, use tokens with appropriate set of policies based on your role in the organization.

To perform all tasks demonstrated in this tutorial, your policy must include the following permissions:

# Work with transform secrets engine
path "transform/*" {
  capabilities = [ "create", "read", "update", "delete", "list" ]
}

# Enable secrets engine
path "sys/mounts/*" {
  capabilities = [ "create", "read", "update", "delete", "list" ]
}

# List enabled secrets engine
path "sys/mounts" {
  capabilities = [ "read", "list" ]
}

If you are not familiar with policies, complete the policies tutorial.

»Setup the Transform secrets engine

Create a role named, mobile-pay which is attached to credit-card transformation. The tokenized value has a fixed maximum time-to-live (TTL) of 24 hours.

Scenario

Enable the transform secrets engine at transform/.
```
$ vault secrets enable transform
```
Create a role named mobile-pay with a transformation named credit-card.
```
$ vault write transform/role/mobile-pay transformations=credit-card

Success! Data written to: transform/role/mobile-pay
```
The mobile-pay role is created.
The role is created but the credit-card transformation does not exist, yet.
Create a transformation named credit-card which sets the generated token's time-to-live (TTL) to 24 hours.
```
$ vault write transform/transformations/tokenization/credit-card \
    allowed_roles=mobile-pay \
    max_ttl=24h
```
The max_ttl is an optional parameter which allows you to control how long the token should stay valid.
NOTE: Set the allowed_roles parameter to a wildcard (*) to allow all roles or with globs at the end for pattern matching (e.g. mobile-*).

Display details about the credit-card transformation.

$ vault read transform/transformations/tokenization/credit-card

Key              Value
---              -----
allowed_roles    [mobile-pay]
mapping_mode     default
max_ttl          24h
templates        <nil>
type             tokenization

Notice that the type is set to tokenization.

»Transform secrets

The Vault client applications must have the following in their policy to perform tokenization transformation using the Transform secrets engine enabled at transform/.

# To request data encoding using any of the roles
# Specify the role name in the path to narrow down the scope
path "transform/encode/mobile-pay" {
   capabilities = [ "update" ]
}

# To request data decoding using any of the roles
# Specify the role name in the path to narrow down the scope
path "transform/decode/mobile-pay" {
   capabilities = [ "update" ]
}

# To validate the token
path "transform/validate/mobile-pay" {
   capabilities = [ "update" ]
}

# To retrieve the metadata belong to the token
path "transform/metadata/mobile-pay" {
   capabilities = [ "update" ]
}

# To check and see if the secret is tokenized
path "transform/tokenized/mobile-pay" {
   capabilities = [ "update" ]
}

Encode a value with the mobile-pay role with some metadata.

$ vault write transform/encode/mobile-pay value=1111-2222-3333-4444 \
    transformation=credit-card \
    ttl=8h \
    metadata="Organization=HashiCorp" \
    metadata="Purpose=Travel" \
    metadata="Type=AMEX"

The ttl value is an optional parameter. Remember that the max_ttl was set to 24 hours when you created the credit-card transformation. You can overwrite that value to make the token's TTL to be shorter.

The output displays the encoded value.

Key              Value
---              -----
encoded_value    Q4tYgFXHxUbU5XZoQsxusAzxhNyVWmCFWxAT8SGkHoYB3VkrmQGXVR

Set the generated token value in a MY_TOKEN environment variable for testing.
Example:
```
$ export MY_TOKEN="Q4tYgFXHxUbU5XZoQsxusAzxhNyVWmCFWxAT8SGkHoYB3VkrmQGXVR"
```

Retrieve the metadata of the token.

$ vault write transform/metadata/mobile-pay value=$MY_TOKEN transformation=credit-card

Key                Value
---                -----
expiration_time    map[nanos:912134000 seconds:1604051798]
metadata           map[Organization:HashiCorp Purpose:Travel Type:AMEX]

Notice that expiration_time is displayed. Since you have overwritten the max_ttl, the ttl is set to 8 hours.

Validate the token value.

$ vault write transform/validate/mobile-pay value=$MY_TOKEN transformation=credit-card

Key      Value
---      -----
valid    true

Validate that the credit card number has been tokenized already.

$ vault write transform/tokenized/mobile-pay value=1111-2222-3333-4444 transformation=credit-card

Key          Value
---          -----
tokenized    true

Retrieve the original plaintext credit card value.

$ vault write transform/decode/mobile-pay transformation=credit-card value=$MY_TOKEN

Key              Value
---              -----
decoded_value    1111-2222-3333-4444

»Setup external token storage

Unlike format preserving encryption (FPE) transformation, tokenization is a stateful procedure to facilitate mapping between tokens and various cryptographic values (one way HMAC of the token, encrypted metadata, etc.) including the encrypted plaintext itself which must be persisted.

At scale, this could put a lot of additional load on the Vault's storage backend. To avoid this, you have an option to use external storage to persist data for tokenization transformation.

NOTE: Currently, supported database for external storage is PostgreSQL.

To demonstrate, run a PostgreSQL database in a Docker container. Create a new transformation named, "passport" which uses this PostgreSQL as its storage rather than using the Vault's storage backend.

Scenario

Run PostgreSQL Docker image in a container.

Start a postgres instance which listens to port 5432, and the superuser (root) password is set to rootpassword.

$ docker run --name postgres -e POSTGRES_USER=root \
     -e POSTGRES_PASSWORD=rootpassword \
     -d -p 5432:5432 postgres

You can verify that the postgres container is running.

$ docker ps

CONTAINER ID        IMAGE            ...         PORTS                    NAMES
befcf913da91        postgres         ...         0.0.0.0:5432->5432/tcp   postgres

Create a new role, "global-id".

$ vault write transform/role/global-id transformations=passport

Create a store which points to the postgres.

$ vault write transform/stores/postgres type=sql supported_transformations=tokenization \
    connection_string='postgresql://{{username}}:{{password}}@localhost/root?sslmode=disable' \
    username=root password=rootpassword

Create a schema in postgres to store tokenization artifacts.

$ vault write transform/stores/postgres/schema transformation_type=tokenization \
    username=root password=rootpassword

Create a new transformation named, "passport" which points to the postgres store.

$ vault write transform/transformations/tokenization/passport \
    allowed_roles=global-id stores=postgres

Open another terminal and connect to the postgres container.

$ docker exec -it postgres bash

Start psql.

$ psql -U root

Check to verify that there is no entry.

$ select * from tokens;

storage_token | key_version | ciphertext | encrypted_metadata | fingerprint | expiration_time
---------------+-------------+------------+--------------------+-------------+-----------------
(0 rows)

Return to the terminal you were running Vault CLI, and encode some test data.

$ vault write transform/encode/global-id transformation=passport \
    value="123456789"

Example output:

Key              Value
---              -----
encoded_value    Q4tYgFXHxUS3PnQLiUnyH2JfGeEZQDFXMMaFXLU6MZfiix1tjqwgNX

Return to the postgres container, and check the data entry.

$ select * from tokens;

Example output:

    storage_token        | key_version |       ciphertext          | encrypted_metadata | ...
--------------------------+-------------+---------------------------+--------------------+-...
\x128aa3c24699...snip... |           1 | \x1ee7cc3505e31...snip... |                    | ...
(1 row)

As you encode more data, the table entry grows.

»Summary

Transformation secrets engine introduced tokenization transformation feature which replaces sensitive data with unique value (token) that are unrelated to the original value in any algorithmic sense. This can help organizations to meet certain industry standards.

If retaining the original data format is important, refer to the Transform Secrets Engine to learn about the format preserving encryption (FPE) transformation.

Infrastructure

Security

Networking

Applications