Tokenize Data with Transform Secrets Engine

»Challenge

When encrypting sensitive data, preservation of the original data format or length may be required to meet certain industry standards such as HIPAA or PCI. To fulfill this requirement, the transform secrets engine performs format preserving encryption (FPE).

However, there are organizations that care more about the irreversibility of the tokenized data and not so much about preserving the original data format. Therefore, the transform secrets engine's FPE transformation may not meet the governance, risk and compliance (GRC) strategy they are looking for due to the use of reversible cryptography to perform FPE.

»Solution

Transform secrets engine has a data transformation method to tokenize sensitive data stored outside of Vault. Tokenization replaces sensitive data with unique values (tokens) that are unrelated to the original value in any algorithmic sense. Therefore, those tokens cannot risk exposing the plaintext satisfying the PCI-DSS guidance.

»Characteristics of the tokenization transformation:

• Non-reversible identification: Protect data pursuant to requirements for data irreversibility (PCI-DSS, GDPR, etc.)

• Integrated Metadata: Supports metadata for identifying data type and purpose

• Extreme scale and performance: Support for performantly managing billions of tokens across clouds as well as on-premise

»Prerequisites

To perform the tasks described in this tutorial, you need to have a Vault Enterprise v1.6 or later with Advanced Data Protection module.

»Policy requirements

To perform all tasks demonstrated in this tutorial, your policy must include the following permissions:

# Work with transform secrets engine
path "transform/*" {
  capabilities = [ "create", "read", "update", "delete", "list" ]
}

# Enable secrets engine
path "sys/mounts/*" {
  capabilities = [ "create", "read", "update", "delete", "list" ]
}

# List enabled secrets engine
path "sys/mounts" {
  capabilities = [ "read", "list" ]
}

# Work with transform secrets engine
path "transform/*" {
  capabilities = [ "create", "read", "update", "delete", "list" ]
}

# Enable secrets engine
path "sys/mounts/*" {
  capabilities = [ "create", "read", "update", "delete", "list" ]
}

# List enabled secrets engine
path "sys/mounts" {
  capabilities = [ "read", "list" ]
}

If you are not familiar with policies, refer to the policies tutorial.

»Setup the Transform secrets engine

Create a role named, mobile-pay which is attached to credit-card transformation. The tokenized value has a fixed maximum time-to-live (TTL) of 24 hours.

»Tokenize secrets

The Vault client applications must have the following in their policy to perform tokenization transformation using the Transform secrets engine enabled at transform/.

# To request data encoding using any of the roles
# Specify the role name in the path to narrow down the scope
path "transform/encode/mobile-pay" {
   capabilities = [ "update" ]
}

# To request data decoding using any of the roles
# Specify the role name in the path to narrow down the scope
path "transform/decode/mobile-pay" {
   capabilities = [ "update" ]
}

# To validate the token
path "transform/validate/mobile-pay" {
   capabilities = [ "update" ]
}

# To retrieve the metadata belong to the token
path "transform/metadata/mobile-pay" {
   capabilities = [ "update" ]
}

# To check and see if the secret is tokenized
path "transform/tokenized/mobile-pay" {
   capabilities = [ "update" ]
}

# To request data encoding using any of the roles
# Specify the role name in the path to narrow down the scope
path "transform/encode/mobile-pay" {
   capabilities = [ "update" ]
}

# To request data decoding using any of the roles
# Specify the role name in the path to narrow down the scope
path "transform/decode/mobile-pay" {
   capabilities = [ "update" ]
}

# To validate the token
path "transform/validate/mobile-pay" {
   capabilities = [ "update" ]
}

# To retrieve the metadata belong to the token
path "transform/metadata/mobile-pay" {
   capabilities = [ "update" ]
}

# To check and see if the secret is tokenized
path "transform/tokenized/mobile-pay" {
   capabilities = [ "update" ]
}

»Convergent tokenization

If you run the command multiple times, you would notice that it returns a different encoded value every time.

Example:

$ vault write transform/encode/mobile-pay value=5555-6666-7777-8888 transformation=credit-card
Key              Value
---              -----
encoded_value    Q4tYgFXHxURKc5zaVDRhi3QT35htvADKtnBpBazpoBZPG373FP1mXs     

$ vault write transform/encode/mobile-pay value=5555-6666-7777-8888 transformation=credit-card
Key              Value
---              -----
encoded_value    Q4tYgFXHxUVpMnf4SSVQZWm1YhAG2eSGvS3dBeDNmm7fkPLnw3JV54     

$ vault write transform/encode/mobile-pay value=5555-6666-7777-8888 transformation=credit-card
Key              Value
---              -----
encoded_value    Q4tYgFXHxURKc5zaVDRhi3QT35htvADKtnBpBazpoBZPG373FP1mXs     

$ vault write transform/encode/mobile-pay value=5555-6666-7777-8888 transformation=credit-card
Key              Value
---              -----
encoded_value    Q4tYgFXHxUVpMnf4SSVQZWm1YhAG2eSGvS3dBeDNmm7fkPLnw3JV54     

In some use cases, you may want to have the same encoded value for a given input so that you can query your database to count the number of entries for a given secret.

Key derivation is supported to allow the same key to be used for multiple purposes by deriving a new key based on a user-supplied context value. In this mode, convergent encryption can optionally be supported, which allows the same input values to produce the same ciphertext.

»Lookup token

When the transformation is configured with convergent encryption, you can look up the tokenized value (token).

»Setup external token storage

Unlike format preserving encryption (FPE) transformation, tokenization is a stateful procedure to facilitate mapping between tokens and various cryptographic values (one way HMAC of the token, encrypted metadata, etc.) including the encrypted plaintext itself which must be persisted.

At scale, this could put a lot of additional load on the Vault's storage backend. To avoid this, you have an option to use external storage to persist data for tokenization transformation.

To demonstrate, run a PostgreSQL database in a Docker container. Create a new transformation named, "passport" which uses this PostgreSQL as its storage rather than using the Vault's storage backend.

Run PostgreSQL Docker image in a container.

Start a postgres instance which listens to port 5432, and the superuser (root) password is set to rootpassword.

$ docker run --name postgres -e POSTGRES_USER=root \
     -e POSTGRES_PASSWORD=rootpassword \
     -d -p 5432:5432 postgres

$ docker run --name postgres -e POSTGRES_USER=root \
     -e POSTGRES_PASSWORD=rootpassword \
     -d -p 5432:5432 postgres

You can verify that the postgres container is running.

$ docker ps

CONTAINER ID        IMAGE            ...         PORTS                    NAMES
befcf913da91        postgres         ...         0.0.0.0:5432->5432/tcp   postgres

$ docker ps

CONTAINER ID        IMAGE            ...         PORTS                    NAMES
befcf913da91        postgres         ...         0.0.0.0:5432->5432/tcp   postgres

»Summary

Transformation secrets engine introduced tokenization transformation feature which replaces sensitive data with unique value (token) that are unrelated to the original value in any algorithmic sense. This can help organizations to meet certain industry standards.

If retaining the original data format is important, refer to the Transform Secrets Engine to learn about the format preserving encryption (FPE) transformation.

»Help and Reference

• Transform Secrets Engine (API)
• Transform Secrets Engine