Transform sensitive data with Vault

50min
|
Enterprise
Vault

Deployment

Note

The Transform secrets engine requires a Vault Enterprise Advanced Data Protection (ADP) license or HCP Vault Dedicated Plus tier cluster.

Challenge

Vault's Transit secrets engine provides encryption service; however, the resulting ciphertext does not preserve the original data format or length.

Think of a scenario where an organization must cryptographically protect personally identifiable information (PII) while preserving the data format and length. For example, the database schema expects a certain character length and/or only allows alphanumeric characters.

The preservation of the original data format or length may be driven by compliance with certain industry standards such as HIPAA or PCI.

Solution

Vault Enterprise 1.4 with Advanced Data Protection module introduced the Transform secrets engine to handle secure data transformation and tokenization against provided secrets. Transformation methods encompass NIST vetted cryptographic standards such as format-preserving encryption (FPE) via FF3-1 to encode your secrets while maintaining the data format and length. In addition, it can also perform pseudonymous transformations of the data through other means, such as masking.

This prevents the need for change in your existing database schemas.

Prerequisites

To perform the tasks described in this tutorial, you need to have:

Vault Enterprise with the Advanced Data Protection module or a HCP Vault Dedicated Plus tier cluster.
jq installed

Policy requirements

For the purpose of this tutorial, you can use root token to work with Vault. The only recommended use for root tokens is for initial setup or in emergencies. As a best practice, use tokens with appropriate set of policies based on your role in the organization.

To perform all tasks demonstrated in this tutorial, your policy must include the following permissions:

# Work with transform secrets engine
path "transform/*" {
  capabilities = [ "create", "read", "update", "delete", "list" ]
}

# Enable secrets engine
path "sys/mounts/*" {
  capabilities = [ "create", "read", "update", "delete", "list" ]
}

# List enabled secrets engine
path "sys/mounts" {
  capabilities = [ "read", "list" ]
}

# Work with transform secrets engine
path "transform/*" {
  capabilities = [ "create", "read", "update", "delete", "list" ]
}

# Enable secrets engine
path "sys/mounts/*" {
  capabilities = [ "create", "read", "update", "delete", "list" ]
}

# List enabled secrets engine
path "sys/mounts" {
  capabilities = [ "read", "list" ]
}

If you are not familiar with policies, complete the policies tutorial.

Lab setup

Export an environment variable with a valid Vault Enterprise license.

$ export VAULT_LICENSE=02MV4UU43BK5....

$ export VAULT_LICENSE=02MV4UU43BK5....

Open a terminal and start a Vault dev server using the Vault Enterprise binary with root as the root token.
```
$ vault server -dev -dev-root-token-id root
```
```
$ vault server -dev -dev-root-token-id root
```
The Vault dev server defaults to running at 127.0.0.1:8200. The server is initialized and unsealed.
Insecure operation
Do not run a Vault dev server in production. This approach starts a Vault server with an in-memory database and runs in an insecure way.

Export an environment variable for the vault CLI to address the Vault server.

$ export VAULT_ADDR=http://127.0.0.1:8200

$ export VAULT_ADDR=http://127.0.0.1:8200

Export an environment variable for the vault CLI to authenticate with the Vault server.
```
$ export VAULT_TOKEN=root
```
```
$ export VAULT_TOKEN=root
```
Note
For these tasks, you can use Vault's root token. However, it is recommended that root tokens are only used for enough initial setup or in emergencies. As a best practice, use an authentication method or token that meets the policy requirements.

The Vault server is ready, you are ready to proceed with the lab.

Transform secrets engine workflow

Transform secrets engine configuration workflow:

Enable the transform secrets engine
Create a role containing the transformations that it can perform
Create an alphabet defining a set of characters to use for format-preserving encryption (FPE) if not using the built-in alphabets.
Create a template defining the rules for value matching if not using the built-in template
Create a transformation to specify the nature of the data manipulation

Alphabets define a set of valid input/output UTF-8 characters to be used when you perform FPE. In this step, you are going to leverage one of the built-in alphabets. Read the create custom alphabets section to learn how to define your own alphabets.

Data transformation templates are constructed of type (regex), pattern (regular expression) and allowed alphabet used in the input value. Currently, regex is the only supported type. The pattern defines the data format pattern. For example, the most credit card numbers would have a pattern that can be expressed as (\d{4})-(\d{4})-(\d{4})-(\d{4}) in regex.

In this step, the use of the builtin/creditcardnumber template is demonstrated. Read the create custom templates section to learn how to define your own templates.

Transformations define the transformation template, tweak source or the masking character to be used to transform the secrets.

Tweak source types:

Source	Description
supplied (default)	User provide the tweak source which must be a base64-encoded 7-digit string
generated	Vault generates and returns the tweak source along with the encoded data. The user must securely store the tweak source which will be needed to decrypt the data
internal	Vault generates a tweak source for the transformation and the same tweak source will be used for every request

Note

Tweak source is only applicable to the FPE transformation.

Create a new transformation

In this section, you are going to:

Enable the transform secrets engine
Create a payments role which includes card-number transformation
Create a card-number transformation which performs format preserving encryption

Enable the transform secrets engine at transform/.

$ vault secrets enable transform
Success! Enabled the transform secrets engine at: transform/

$ vault secrets enable transform
Success! Enabled the transform secrets engine at: transform/

Create a role named "payments" with "card-number" transformation attached which you will create next.

$ vault write transform/role/payments transformations=card-number
Success! Data written to: transform/role/payments

$ vault write transform/role/payments transformations=card-number
Success! Data written to: transform/role/payments

List the existing roles.

$ vault list transform/role
Keys
----
payments

$ vault list transform/role
Keys
----
payments

Create a transformation named "card-number" which will be used to transform credit card numbers. This uses the built-in template builtin/creditcardnumber to perform format-preserving encryption (FPE). The allowed role to use this transformation is payments, which you just created.
```
$ vault write transform/transformations/fpe/card-number \
    template="builtin/creditcardnumber" \
    tweak_source=internal \
    allowed_roles=payments
```
```
$ vault write transform/transformations/fpe/card-number \
    template="builtin/creditcardnumber" \
    tweak_source=internal \
    allowed_roles=payments
```
Example Output:
```
Success! Data written to: transform/transformations/fpe/card-number
```
```
Success! Data written to: transform/transformations/fpe/card-number
```
Note
The allowed_roles parameter can be set to a wildcard (*) instead of listing role names. Also, the role name can be expressed using globs at the end for pattern matching (e.g. pay*).
You will learn how to define your own template in the Create custom templates section.

List the existing transformations.

$ vault list transform/transformations/fpe
Keys
----
card-number

$ vault list transform/transformations/fpe
Keys
----
card-number

View the details of the newly created card-number transformation.

$ vault read transform/transformations/fpe/card-number
Key                 Value
---                 -----
allowed_roles       [payments]
deletion_allowed    false
template            builtin/creditcardnumber
templates           [builtin/creditcardnumber]
tweak_source        internal
type                fpe

$ vault read transform/transformations/fpe/card-number
Key                 Value
---                 -----
allowed_roles       [payments]
deletion_allowed    false
template            builtin/creditcardnumber
templates           [builtin/creditcardnumber]
tweak_source        internal
type                fpe

Note

Unless called out with an Example output block, API calls do not produce output.

Enable transform secrets engine using /sys/mounts endpoint.

$ curl --header "X-Vault-Token: $VAULT_TOKEN" \
      --request POST \
      --data '{"type":"transform"}' \
      $VAULT_ADDR/v1/sys/mounts/transform

$ curl --header "X-Vault-Token: $VAULT_TOKEN" \
      --request POST \
      --data '{"type":"transform"}' \
      $VAULT_ADDR/v1/sys/mounts/transform

Create a role named payments with a transformation named card-number.

$ curl --header "X-Vault-Token: $VAULT_TOKEN" \
      --request POST \
      --data '{"transformations": ["card-number"]}' \
      $VAULT_ADDR/v1/transform/role/payments

$ curl --header "X-Vault-Token: $VAULT_TOKEN" \
      --request POST \
      --data '{"transformations": ["card-number"]}' \
      $VAULT_ADDR/v1/transform/role/payments

The payments role is created.

Display all the roles.

$ curl --header "X-Vault-Token: $VAULT_TOKEN" \
      --request LIST \
      $VAULT_ADDR/v1/transform/role | jq ".data"

$ curl --header "X-Vault-Token: $VAULT_TOKEN" \
      --request LIST \
      $VAULT_ADDR/v1/transform/role | jq ".data"

Example output:

{
  "keys": [
    "payments"
  ]
}

{
  "keys": [
    "payments"
  ]
}

Create a transformation API request payload.
```
$ tee payload.json <<EOF
{
  "template": "builtin/creditcardnumber",
  "tweak_source": "internal",
  "allowed_roles": [ "payments" ]
}
EOF
```
```
$ tee payload.json <<EOF
{
  "template": "builtin/creditcardnumber",
  "tweak_source": "internal",
  "allowed_roles": [ "payments" ]
}
EOF
```
The template value specifies a built-in builtin/creditcardnumber template to perform format-preserving encryption (FPE). The allowed_roles is limited to the role named payments.
Note
The allowed_roles parameter can be set to a wildcard (*) instead of listing role names. Also, the role name can be expressed using globs at the end for pattern matching (e.g. pay*).
You will learn how to define your own template in the Create custom templates section.

Create a transformation named card-number.

$ curl --header "X-Vault-Token: $VAULT_TOKEN" \
      --request POST \
      --data @payload.json \
      $VAULT_ADDR/v1/transform/transformations/fpe/card-number

$ curl --header "X-Vault-Token: $VAULT_TOKEN" \
      --request POST \
      --data @payload.json \
      $VAULT_ADDR/v1/transform/transformations/fpe/card-number

Display all the transformations.

$ curl --header "X-Vault-Token: $VAULT_TOKEN" \
        --request LIST \
        $VAULT_ADDR/v1/transform/transformations/fpe | jq ".data"

$ curl --header "X-Vault-Token: $VAULT_TOKEN" \
        --request LIST \
        $VAULT_ADDR/v1/transform/transformations/fpe | jq ".data"

Example output:

{
  "keys": [
    "card-number"
  ]
}

{
  "keys": [
    "card-number"
  ]
}

Display details about the card-number transformation

$ curl --header "X-Vault-Token: $VAULT_TOKEN" \
      $VAULT_ADDR/v1/transform/transformations/fpe/card-number \
      | jq ".data"

$ curl --header "X-Vault-Token: $VAULT_TOKEN" \
      $VAULT_ADDR/v1/transform/transformations/fpe/card-number \
      | jq ".data"

Example output:

{
  "allowed_roles": [
    "payments"
  ],
  "deletion_allowed": false,
  "template": "builtin/creditcardnumber",
  "templates": [
    "builtin/creditcardnumber"
  ],
  "tweak_source": "internal",
  "type": "fpe"
}

{
  "allowed_roles": [
    "payments"
  ],
  "deletion_allowed": false,
  "template": "builtin/creditcardnumber",
  "templates": [
    "builtin/creditcardnumber"
  ],
  "tweak_source": "internal",
  "type": "fpe"
}

Open a web browser, then navigate to the Vault UI and login.
Click Enable new engine and select the Transform radio button.
Click Next.
Click Enable Engine to accept the default path, transform.
Click Create transformation from the Transformations tab.
Enter the following information:
- Name - Enter card-number
- Type - Select Format Preserving Encryption
- Tweak - source Select internal
- Template Name - Start typing creditcard and select builtin/creditcardnumber.
- Allowed roles - Enter payments. This role does not exist yet, so Vault will create it and attach the transformation to it.
You will learn how to define your own template in the Create custom templates section.
Note
The allowed_roles parameter can be set to a wildcard (*) instead of listing role names. Also, the role name can be expressed using globs at the end for pattern matching (e.g. pay*).
Click Create Transformation. This displays the details page of the transformation including encode and decode commands that can copy to test the transformation.

Transform secrets

The Vault client applications must have the following in their policy to perform data encoding and decoding using the Transform secrets engine enabled at transform/.

# To request data encoding using any of the roles
# Specify the role name in the path to narrow down the scope
path "transform/encode/*" {
   capabilities = [ "update" ]
}

# To request data decoding using any of the roles
# Specify the role name in the path to narrow down the scope
path "transform/decode/*" {
   capabilities = [ "update" ]
}

# To request data encoding using any of the roles
# Specify the role name in the path to narrow down the scope
path "transform/encode/*" {
   capabilities = [ "update" ]
}

# To request data decoding using any of the roles
# Specify the role name in the path to narrow down the scope
path "transform/decode/*" {
   capabilities = [ "update" ]
}

Encode a value with the payments role.

$ ENCODED_OUTPUT=$(vault write transform/encode/payments \
     value=1111-2222-3333-4444 \
     -format=json | jq -r '.data | .encoded_value') \
     && echo encoded_value: $ENCODED_OUTPUT

$ ENCODED_OUTPUT=$(vault write transform/encode/payments \
     value=1111-2222-3333-4444 \
     -format=json | jq -r '.data | .encoded_value') \
     && echo encoded_value: $ENCODED_OUTPUT

Example output:

encoded_value: 9474-2096-7736-0783

encoded_value: 9474-2096-7736-0783

Decode the value encoded with payments role where the value is set to the returned encoded_value.

$ vault write transform/decode/payments \
        value=$ENCODED_OUTPUT

$ vault write transform/decode/payments \
        value=$ENCODED_OUTPUT

Example output:

Key              Value
---              -----
decoded_value    1111-2222-3333-4444

Key              Value
---              -----
decoded_value    1111-2222-3333-4444

Encode a value with the payments role.

$ ENCODED_OUTPUT=$(curl --header "X-Vault-Token: $VAULT_TOKEN" \
      --request POST \
      --data '{"value": "1111-2222-3333-4444"}' \
      $VAULT_ADDR/v1/transform/encode/payments | jq -r ".data.encoded_value") \
      && echo encoded_value: $ENCODED_OUTPUT

$ ENCODED_OUTPUT=$(curl --header "X-Vault-Token: $VAULT_TOKEN" \
      --request POST \
      --data '{"value": "1111-2222-3333-4444"}' \
      $VAULT_ADDR/v1/transform/encode/payments | jq -r ".data.encoded_value") \
      && echo encoded_value: $ENCODED_OUTPUT

Example output:

encoded_value: "3330-9570-2888-3352"

encoded_value: "3330-9570-2888-3352"

Decode the value encoded with payments role.

$ curl --header "X-Vault-Token: $VAULT_TOKEN" \
      --request POST \
      --data '{"value": "3330-9570-2888-3352"}' \
      $VAULT_ADDR/v1/transform/decode/payments | jq ".data"

$ curl --header "X-Vault-Token: $VAULT_TOKEN" \
      --request POST \
      --data '{"value": "3330-9570-2888-3352"}' \
      $VAULT_ADDR/v1/transform/decode/payments | jq ".data"

Example output:

{
  "decoded_value": "1111-2222-3333-4444"
}

{
  "decoded_value": "1111-2222-3333-4444"
}

Note

Refer to the batch input processing section for encoding and decoding multiple secrets in a single API call.

The transformation detail page displays a sample encode and decode commands.
Copy the encode command.
Click the Vault CLI shell icon (>_) to open a command shell. Paste in the copied command with some test data.
Example:
```
$ vault write transform/encode/payments value="1111-2222-3333-4444"
```
```
$ vault write transform/encode/payments value="1111-2222-3333-4444"
```
You can omit the transformation=card-number since the payments role only has one transformation attached.
Press Enter to execute the command.

To decode the value, set the value to be the returned encoded_value and invoke the transform/decode/payments endpoint.

Example:

$ vault write transform/decode/payments value="3501-8042-9628-0343"

$ vault write transform/decode/payments value="3501-8042-9628-0343"

ui-vault-vault-shell-decode-value

Close the Vault UI shell and click transform in the breadcrumb menu to return to the main Transform secrets engine configuration page.

Create custom templates

Templates define the data format patterns that you wish to keep while transforming the secrets. In this section, you are going to create a transformation template which encodes British passport numbers.

The passport number on a British passport is a pattern consisting of a 9-digit numeric value which can be expressed using regular expression as (\d{9}). The parentheses tell Vault to encode all values grouped within; therefore, (\d{9}) will encode the entire passport number.

If you want to encode the last 7 digits leaving the first two numbers unchanged, the expression should be \d{2}(\d{7}).

Display all the exiting templates.

$ vault list transform/template

Keys
----
builtin/creditcardnumber
builtin/socialsecuritynumber

$ vault list transform/template

Keys
----
builtin/creditcardnumber
builtin/socialsecuritynumber

Create a template named uk-passport-tmpl.

$ vault write transform/template/uk-passport-tmpl \
    type=regex \
    pattern="(\d{9})" \
    alphabet=builtin/numeric

$ vault write transform/template/uk-passport-tmpl \
    type=regex \
    pattern="(\d{9})" \
    alphabet=builtin/numeric

Example output:

Success! Data written to: transform/template/uk-passport-tmpl

Success! Data written to: transform/template/uk-passport-tmpl

This template uses the built-in alphabet, builtin/numeric.

Create a transformation named uk-passport with the uk-passport-tmpl template.

$ vault write transform/transformations/fpe/uk-passport \
    template=uk-passport-tmpl \
    tweak_source=internal \
    allowed_roles='*'

$ vault write transform/transformations/fpe/uk-passport \
    template=uk-passport-tmpl \
    tweak_source=internal \
    allowed_roles='*'

Example output:

Success! Data written to: transform/transformations/fpe/uk-passport

Success! Data written to: transform/transformations/fpe/uk-passport

Update the payments role to include the uk-passport transformation.

$ vault write transform/role/payments transformations=card-number,uk-passport
Success! Data written to: transform/role/payments

$ vault write transform/role/payments transformations=card-number,uk-passport
Success! Data written to: transform/role/payments

The payments role has two transformations. Future requests to encode/decode require that the specific transformation is provided.

Encode a value with the payments role with the uk-passport.

$ ENCODED_OUTPUT=$(vault write transform/encode/payments value="123456789" \
    transformation=uk-passport -format=json | jq -r '.data | .encoded_value') \
    && echo encoded_value: $ENCODED_OUTPUT

$ ENCODED_OUTPUT=$(vault write transform/encode/payments value="123456789" \
    transformation=uk-passport -format=json | jq -r '.data | .encoded_value') \
    && echo encoded_value: $ENCODED_OUTPUT

Example output:

encoded_value: 128151714

encoded_value: 128151714

Create an API request payload that describes the template.

$ tee payload-template.json <<EOF
{
  "type": "regex",
  "pattern": "(\\\d{9})",
  "alphabet": "builtin/numeric"
}
EOF

$ tee payload-template.json <<EOF
{
  "type": "regex",
  "pattern": "(\\\d{9})",
  "alphabet": "builtin/numeric"
}
EOF

This template uses the built-in alphabet, builtin/numeric.

Create a template named uk-passport-tmpl.

$ curl --header "X-Vault-Token: $VAULT_TOKEN" \
     --request POST \
     --data @payload-template.json \
     $VAULT_ADDR/v1/transform/template/uk-passport-tmpl

$ curl --header "X-Vault-Token: $VAULT_TOKEN" \
     --request POST \
     --data @payload-template.json \
     $VAULT_ADDR/v1/transform/template/uk-passport-tmpl

The transformation template is created.

Create an API request payload that describes a transformation.

$ tee payload.json <<EOF
{
  "template": "uk-passport-tmpl",
  "tweak_source": "internal",
  "allowed_roles": ["*"]
}
EOF

$ tee payload.json <<EOF
{
  "template": "uk-passport-tmpl",
  "tweak_source": "internal",
  "allowed_roles": ["*"]
}
EOF

The payload uses the template uk-passport-tmpl.

Create a transformation named uk-passport.

$ curl --header "X-Vault-Token: $VAULT_TOKEN" \
     --request POST \
     --data @payload.json \
     $VAULT_ADDR/v1/transform/transformations/fpe/uk-passport

$ curl --header "X-Vault-Token: $VAULT_TOKEN" \
     --request POST \
     --data @payload.json \
     $VAULT_ADDR/v1/transform/transformations/fpe/uk-passport

Update the payments role to include the uk-passport transformation.

$ curl --header "X-Vault-Token: $VAULT_TOKEN" \
     --request POST \
     --data '{"transformations": ["uk-passport", "card-number"]}' \
     $VAULT_ADDR/v1/transform/role/payments

$ curl --header "X-Vault-Token: $VAULT_TOKEN" \
     --request POST \
     --data '{"transformations": ["uk-passport", "card-number"]}' \
     $VAULT_ADDR/v1/transform/role/payments

The payments role has two transformations. Future requests to encode/decode require that the specific transformation is provided.

Create an API request payload with a value and the uk-passport transformation.

$ tee input.json <<EOF
{
  "value": "123456789",
  "transformation": "uk-passport"
}
EOF

$ tee input.json <<EOF
{
  "value": "123456789",
  "transformation": "uk-passport"
}
EOF

Encode a value with the payments role with the uk-passport transformation.

$ curl --header "X-Vault-Token: $VAULT_TOKEN" \
     --request POST \
     --data @input.json \
     $VAULT_ADDR/v1/transform/encode/payments | jq ".data"

$ curl --header "X-Vault-Token: $VAULT_TOKEN" \
     --request POST \
     --data @input.json \
     $VAULT_ADDR/v1/transform/encode/payments | jq ".data"

Example output:

{
  "encoded_value": "128151714"
}

{
  "encoded_value": "128151714"
}

Select the Templates tab. Existing templates are listed.
In the Templates tab, click Create template.
Enter the following information:
- Name - Enter uk-passport-tmpl
- Pattern - Enter (\d{9})
- Alphabet - Select builtin/numeric
Click Create template to complete.
Click transform in the breadcrumb menu to return to the Templates configuration page.
Select the Transformations tab and click Create transformation.
Enter the following information:
- Name - Enter uk-passport
- Type - Select Format Preserving Encryption (FPE)
- Tweak source - Select internal
- Template Name - Choose the uk-passport-tmpl you created earlier
- Allowed roles - Select payments
Click Create transformation. The payments role has two transformations. Future requests to encode/decode require that the specific transformation is provided by passing the transformation parameter.

Click the Vault CLI shell icon (>_) to open a command shell. Paste in the copied command with some test data and the transformation parameter.

$ vault write transform/encode/payments value="123456789" transformation=uk-passport

$ vault write transform/encode/payments value="123456789" transformation=uk-passport

Click transform in the breadcrumb menu to return to the main Transform secrets engine configuration page.

Note

Remember that you must specify which transformation to use when you send an encode request since the payments role has two transformations associated with it.

Advanced handling

This section walks you through the new template features introduced in Vault Enterprise v1.9:

Encoding customization

In this section, you are going to create a transformation template which encodes Social Security numbers that may have an optional SSN: or ssn: prefix, and which are optionally separated by dashes or spaces.

A United States Social Security number is a 9-digit number, commonly written using a 3-2-4 digit pattern which can be expressed using the regular expression (\d{3})[- ]?(\d{2})[- ]?(\d{4}). The optional prefix can be expressed using the regular expression (?:SSN[: ]?|ssn[: ]?)?. The use of non-capturing groups tells Vault not to encode the prefix if it is present.

You will use the new encode_format field to specify what the encoded output should look like. In the value for encode_format, variables representing the capture groups of pattern are used to lay out the result. The variables are in the form of $1, $2, etc…, one for each of the capture groups in pattern; and in the form of ${name} or $name for the named capture groups. For more detailed information see the Go regexp.Expand documentation.

When a template has a value for encode_format, it will always be used. Make sure that the resulting encoded output can be matched by pattern, otherwise decoding will not be possible.

Create a template named us-ssn-tmpl.

$  vault write transform/template/us-ssn-tmpl \
     type=regex \
     pattern='(?:SSN[: ]?|ssn[: ]?)?(\d{3})[- ]?(\d{2})[- ]?(\d{4})' \
     encode_format='$1-$2-$3' \
     alphabet=builtin/numeric

$  vault write transform/template/us-ssn-tmpl \
     type=regex \
     pattern='(?:SSN[: ]?|ssn[: ]?)?(\d{3})[- ]?(\d{2})[- ]?(\d{4})' \
     encode_format='$1-$2-$3' \
     alphabet=builtin/numeric

Example output:

Success! Data written to: transform/template/us-ssn-tmpl

Success! Data written to: transform/template/us-ssn-tmpl

Create a transformation named us-ssn with the us-ssn-tmpl template.

$ vault write transform/transformations/fpe/us-ssn \
    template=us-ssn-tmpl \
    tweak_source=internal \
    allowed_roles='*'

$ vault write transform/transformations/fpe/us-ssn \
    template=us-ssn-tmpl \
    tweak_source=internal \
    allowed_roles='*'

Example output:

Success! Data written to: transform/transformations/fpe/us-ssn

Success! Data written to: transform/transformations/fpe/us-ssn

Update the payments role to include the us-ssn transformation.

$ vault write transform/role/payments transformations=card-number,uk-passport,us-ssn
Success! Data written to: transform/role/payments

$ vault write transform/role/payments transformations=card-number,uk-passport,us-ssn
Success! Data written to: transform/role/payments

Validation

Encode values with the payments role with the us-ssn transformation.

$ ENCODED_OUTPUT=$(vault write transform/encode/payments value="123-45-6789" \
    transformation=us-ssn \
    -format=json | jq -r '.data | .encoded_value') \
    && echo encoded_value: $ENCODED_OUTPUT

$ ENCODED_OUTPUT=$(vault write transform/encode/payments value="123-45-6789" \
    transformation=us-ssn \
    -format=json | jq -r '.data | .encoded_value') \
    && echo encoded_value: $ENCODED_OUTPUT

Example output:

encoded_value: 489-82-2140

encoded_value: 489-82-2140

Try encoding value that starts with SSN.

$ vault write transform/encode/payments value="SSN:$ENCODED_OUTPUT" \
    transformation=us-ssn

$ vault write transform/encode/payments value="SSN:$ENCODED_OUTPUT" \
    transformation=us-ssn

Example output:

Key              Value
---              -----
encoded_value    766-52-7759

Key              Value
---              -----
encoded_value    766-52-7759

Try encoding value that starts with ssn.

$  vault write transform/encode/payments value="ssn:$ENCODED_OUTPUT" \
     transformation=us-ssn

$  vault write transform/encode/payments value="ssn:$ENCODED_OUTPUT" \
     transformation=us-ssn

Example output:

Key              Value
---              -----
encoded_value    766-52-7759

Key              Value
---              -----
encoded_value    766-52-7759

Decode the value encoded with the payments role with the us-ssn transformation where the value is set to the encoded_value.

$ vault write transform/decode/payments value="$ENCODED_OUTPUT" \
    transformation=us-ssn

$ vault write transform/decode/payments value="$ENCODED_OUTPUT" \
    transformation=us-ssn

Example output:

Key              Value
---              -----
decoded_value    123-45-6789

Key              Value
---              -----
decoded_value    123-45-6789

Create an API request payload that describes the template.

$ tee payload-template.json <<EOF
{
  "type": "regex",
  "pattern": "(?:SSN[: ]?|ssn[: ]?)?(\\\\d{3})[- ]?(\\\\d{2})[- ]?(\\\\d{4})",
  "encode_format": "\$1-\$2-\$3",
  "alphabet": "builtin/numeric"
}
EOF

$ tee payload-template.json <<EOF
{
  "type": "regex",
  "pattern": "(?:SSN[: ]?|ssn[: ]?)?(\\\\d{3})[- ]?(\\\\d{2})[- ]?(\\\\d{4})",
  "encode_format": "\$1-\$2-\$3",
  "alphabet": "builtin/numeric"
}
EOF

Create a template named us-ssn-tmpl.

$ curl --header "X-Vault-Token: $VAULT_TOKEN" \
      --request POST \
      --data @payload-template.json \
      $VAULT_ADDR/v1/transform/template/us-ssn-tmpl

$ curl --header "X-Vault-Token: $VAULT_TOKEN" \
      --request POST \
      --data @payload-template.json \
      $VAULT_ADDR/v1/transform/template/us-ssn-tmpl

Create an API request payload that describes a transformation.

$ tee payload.json <<EOF
{
  "template":"us-ssn-tmpl",
  "tweak_source":"internal",
  "allowed_roles":"*"
}
EOF

$ tee payload.json <<EOF
{
  "template":"us-ssn-tmpl",
  "tweak_source":"internal",
  "allowed_roles":"*"
}
EOF

Create a transformation named us-ssn with the us-ssn-tmpl template.

$ curl --header "X-Vault-Token: $VAULT_TOKEN" \
     --request POST \
     --data @payload.json \
     $VAULT_ADDR/v1/transform/transformations/fpe/us-ssn

$ curl --header "X-Vault-Token: $VAULT_TOKEN" \
     --request POST \
     --data @payload.json \
     $VAULT_ADDR/v1/transform/transformations/fpe/us-ssn

Update the payments role to include the us-ssn transformation.

$ curl --header "X-Vault-Token: $VAULT_TOKEN" \
     --request POST \
     --data '{"transformations": ["uk-passport", "card-number", "us-ssn"]}' \
     $VAULT_ADDR/v1/transform/role/payments

$ curl --header "X-Vault-Token: $VAULT_TOKEN" \
     --request POST \
     --data '{"transformations": ["uk-passport", "card-number", "us-ssn"]}' \
     $VAULT_ADDR/v1/transform/role/payments

Validation

Encode values with the payments role with the us-ssn transformation.

$ ENCODED_OUTPUT=$(curl --header "X-Vault-Token: $VAULT_TOKEN" \
     --request POST \
     --data '{"value": "123-45-6789", "transformation": "us-ssn"}' \
     $VAULT_ADDR/v1/transform/encode/payments | jq ".data.encoded_value") \
     && echo encoded_value: $ENCODED_OUTPUT

$ ENCODED_OUTPUT=$(curl --header "X-Vault-Token: $VAULT_TOKEN" \
     --request POST \
     --data '{"value": "123-45-6789", "transformation": "us-ssn"}' \
     $VAULT_ADDR/v1/transform/encode/payments | jq ".data.encoded_value") \
     && echo encoded_value: $ENCODED_OUTPUT

Example output:

encoded_value: "766-52-7759"

encoded_value: "766-52-7759"

Try encoding value that starts with SSN.

$ curl --header "X-Vault-Token: $VAULT_TOKEN" \
     --request POST \
     --data '{"value": "SSN:123 45 6789", "transformation": "us-ssn"}' \
     $VAULT_ADDR/v1/transform/encode/payments | jq ".data"

$ curl --header "X-Vault-Token: $VAULT_TOKEN" \
     --request POST \
     --data '{"value": "SSN:123 45 6789", "transformation": "us-ssn"}' \
     $VAULT_ADDR/v1/transform/encode/payments | jq ".data"

Example output:

{
  "encoded_value": "766-52-7759"
}

{
  "encoded_value": "766-52-7759"
}

Try encoding value that starts with ssn.

$ curl --header "X-Vault-Token: $VAULT_TOKEN" \
     --request POST \
     --data '{"value": "ssn:123456789", "transformation": "us-ssn"}' \
     $VAULT_ADDR/v1/transform/encode/payments | jq ".data"

$ curl --header "X-Vault-Token: $VAULT_TOKEN" \
     --request POST \
     --data '{"value": "ssn:123456789", "transformation": "us-ssn"}' \
     $VAULT_ADDR/v1/transform/encode/payments | jq ".data"

Example output:

{
  "encoded_value": "766-52-7759"
}

{
  "encoded_value": "766-52-7759"
}

Decode the value encoded with the payments role with the us-ssn transformation where value is set to the encoded_value.

$ curl --header "X-Vault-Token: $VAULT_TOKEN" \
      --request POST \
      --data '{"value": '"$ENCODED_OUTPUT"', "transformation": "us-ssn"}' \
      $VAULT_ADDR/v1/transform/decode/payments | jq ".data"

$ curl --header "X-Vault-Token: $VAULT_TOKEN" \
      --request POST \
      --data '{"value": '"$ENCODED_OUTPUT"', "transformation": "us-ssn"}' \
      $VAULT_ADDR/v1/transform/decode/payments | jq ".data"

Example output:

{
  "decoded_value": "123-45-6789"
}

{
  "decoded_value": "123-45-6789"
}

Decoding customization

When running Vault Enterprise v1.9 or later, you can specify one or more optional formats to use during decoding. In this section, you are going to modify the template us-ssn-tmpl, created in the previous section, to add two decoding formats: one to decode values separated by spaces, and the second to decode only the last four digits.

Like encode_format, decode_formats have variables representing the capture groups of pattern to lay out the decoded output. Specifying a decode format is optional, and it is specified as part of the path when performing the decode operation.

Create a template named us-ssn-tmpl.

$  vault write transform/template/us-ssn-tmpl \
     type=regex \
     pattern='(?:SSN[: ]?|ssn[: ]?)?(\d{3})[- ]?(\d{2})[- ]?(\d{4})' \
     encode_format='$1-$2-$3' \
     decode_formats=space-separated='$1 $2 $3' \
     decode_formats=last-four='*** ** $3' \
     alphabet=builtin/numeric

$  vault write transform/template/us-ssn-tmpl \
     type=regex \
     pattern='(?:SSN[: ]?|ssn[: ]?)?(\d{3})[- ]?(\d{2})[- ]?(\d{4})' \
     encode_format='$1-$2-$3' \
     decode_formats=space-separated='$1 $2 $3' \
     decode_formats=last-four='*** ** $3' \
     alphabet=builtin/numeric

Example output:

Success! Data written to: transform/template/us-ssn-tmpl

Success! Data written to: transform/template/us-ssn-tmpl

Create a transformation named us-ssn with the us-ssn-tmpl template.

$ vault write transform/transformations/fpe/us-ssn \
    template=us-ssn-tmpl \
    tweak_source=internal \
    allowed_roles='*'

$ vault write transform/transformations/fpe/us-ssn \
    template=us-ssn-tmpl \
    tweak_source=internal \
    allowed_roles='*'

Example output:

Success! Data written to: transform/transformations/fpe/us-ssn

Success! Data written to: transform/transformations/fpe/us-ssn

Update the payments role to include the us-ssn transformation.

$ vault write transform/role/payments transformations=card-number,uk-passport,us-ssn
Success! Data written to: transform/role/payments

$ vault write transform/role/payments transformations=card-number,uk-passport,us-ssn
Success! Data written to: transform/role/payments

Encode values with the payments role with the us-ssn transformation.

$ ENCODED_OUTPUT=$(vault write transform/encode/payments value="123-45-6789" \
    transformation=us-ssn \
    -format=json | jq -r '.data | .encoded_value') \
    && echo encoded_value: $ENCODED_OUTPUT

$ ENCODED_OUTPUT=$(vault write transform/encode/payments value="123-45-6789" \
    transformation=us-ssn \
    -format=json | jq -r '.data | .encoded_value') \
    && echo encoded_value: $ENCODED_OUTPUT

Example output:

encoded_value: 489-82-2140

encoded_value: 489-82-2140

Decode values with the payments role, the us-ssn transformation and the space-separated decoding format.

$ vault write transform/decode/payments/space-separated value="$ENCODED_OUTPUT" \
     transformation=us-ssn

$ vault write transform/decode/payments/space-separated value="$ENCODED_OUTPUT" \
     transformation=us-ssn

Example output:

Key              Value
---              -----
decoded_value    123 45 6789

Key              Value
---              -----
decoded_value    123 45 6789

Decode values with the payments role, the us-ssn transformation and the last-four decoding format.

$ vault write transform/decode/payments/last-four value="$ENCODED_OUTPUT" \
     transformation=us-ssn

$ vault write transform/decode/payments/last-four value="$ENCODED_OUTPUT" \
     transformation=us-ssn

Example output:

Key              Value
---              -----
decoded_value    *** ** 6789

Key              Value
---              -----
decoded_value    *** ** 6789

Create an API request payload that describes the template.

$ tee payload-template.json <<EOF
{
  "type": "regex",
  "pattern": "(?:SSN[: ]?|ssn[: ]?)?(\\\\d{3})[- ]?(\\\\d{2})[- ]?(\\\\d{4})",
  "encode_format": "\$1-\$2-\$3",
  "decode_formats": {
    "space-separated": "\$1 \$2 \$3",
    "last-four": "*** ** \$3"
  },
  "alphabet": "builtin/numeric"
}
EOF

$ tee payload-template.json <<EOF
{
  "type": "regex",
  "pattern": "(?:SSN[: ]?|ssn[: ]?)?(\\\\d{3})[- ]?(\\\\d{2})[- ]?(\\\\d{4})",
  "encode_format": "\$1-\$2-\$3",
  "decode_formats": {
    "space-separated": "\$1 \$2 \$3",
    "last-four": "*** ** \$3"
  },
  "alphabet": "builtin/numeric"
}
EOF

Create a template named us-ssn-tmpl.

$ curl --header "X-Vault-Token: $VAULT_TOKEN" \
      --request POST \
      --data @payload-template.json \
      $VAULT_ADDR/v1/transform/template/us-ssn-tmpl

$ curl --header "X-Vault-Token: $VAULT_TOKEN" \
      --request POST \
      --data @payload-template.json \
      $VAULT_ADDR/v1/transform/template/us-ssn-tmpl

Create an API request payload that describes a transformation.

$ tee payload.json <<EOF
{
  "template":"us-ssn-tmpl",
  "tweak_source":"internal",
  "allowed_roles":"*"
}
EOF

$ tee payload.json <<EOF
{
  "template":"us-ssn-tmpl",
  "tweak_source":"internal",
  "allowed_roles":"*"
}
EOF

Create a transformation named us-ssn with the us-ssn-tmpl template.

$ curl --header "X-Vault-Token: $VAULT_TOKEN" \
     --request POST \
     --data @payload.json \
     $VAULT_ADDR/v1/transform/transformations/fpe/us-ssn

$ curl --header "X-Vault-Token: $VAULT_TOKEN" \
     --request POST \
     --data @payload.json \
     $VAULT_ADDR/v1/transform/transformations/fpe/us-ssn

Update the payments role to include the us-ssn transformation.

$ curl --header "X-Vault-Token: $VAULT_TOKEN" \
     --request POST \
     --data '{"transformations": ["uk-passport", "card-number", "us-ssn"]}' \
     $VAULT_ADDR/v1/transform/role/payments

$ curl --header "X-Vault-Token: $VAULT_TOKEN" \
     --request POST \
     --data '{"transformations": ["uk-passport", "card-number", "us-ssn"]}' \
     $VAULT_ADDR/v1/transform/role/payments

Encode values with the payments role with the us-ssn transformation.

$ ENCODED_OUTPUT=$(curl --header "X-Vault-Token: $VAULT_TOKEN" \
     --request POST \
     --data '{"value": "123-45-6789", "transformation": "us-ssn"}' \
     $VAULT_ADDR/v1/transform/encode/payments | jq ".data.encoded_value") \
     && echo encoded_value: $ENCODED_OUTPUT

$ ENCODED_OUTPUT=$(curl --header "X-Vault-Token: $VAULT_TOKEN" \
     --request POST \
     --data '{"value": "123-45-6789", "transformation": "us-ssn"}' \
     $VAULT_ADDR/v1/transform/encode/payments | jq ".data.encoded_value") \
     && echo encoded_value: $ENCODED_OUTPUT

Example output:

encoded_value: "766-52-7759"

encoded_value: "766-52-7759"

Decode values with the payments role, the us-ssn transformation and the space-separated decoding format.

$ curl --header "X-Vault-Token: $VAULT_TOKEN" \
     --request POST \
     --data '{"value": '"$ENCODED_OUTPUT"', "transformation": "us-ssn"}' \
     $VAULT_ADDR/v1/transform/decode/payments/space-separated | jq ".data"

$ curl --header "X-Vault-Token: $VAULT_TOKEN" \
     --request POST \
     --data '{"value": '"$ENCODED_OUTPUT"', "transformation": "us-ssn"}' \
     $VAULT_ADDR/v1/transform/decode/payments/space-separated | jq ".data"

Example output:

{
  "encoded_value": "123 45 6789"
}

{
  "encoded_value": "123 45 6789"
}

Decode values with the payments role, the us-ssn transformation and the last-four decoding format.

$ curl --header "X-Vault-Token: $VAULT_TOKEN" \
     --request POST \
     --data '{"value": '"$ENCODED_OUTPUT"', "transformation": "us-ssn"}' \
     $VAULT_ADDR/v1/transform/decode/payments/last-four | jq ".data"

$ curl --header "X-Vault-Token: $VAULT_TOKEN" \
     --request POST \
     --data '{"value": '"$ENCODED_OUTPUT"', "transformation": "us-ssn"}' \
     $VAULT_ADDR/v1/transform/decode/payments/last-four | jq ".data"

Example output:

{
  "encoded_value": "*** ** 6789"
}

{
  "encoded_value": "*** ** 6789"
}

Access control

As the decode format is part of the path of the write operation during decoding, Vault policies can be used to control access to them.

# Allow decoding using any of the decode formats
path "transform/decode/us-ssn/*"
{
    capabilities = ["update"]
}
# Allow decoding using only the last-four decode format
path "transform/decode/us-ssn/last-four"
{
    capabilities = ["update"]
}
# Allow decoding without specifying a decode format
path "transform/decode/us-ssn"
{
    capabilities = ["update"]
}

# Allow decoding using any of the decode formats
path "transform/decode/us-ssn/*"
{
    capabilities = ["update"]
}
# Allow decoding using only the last-four decode format
path "transform/decode/us-ssn/last-four"
{
    capabilities = ["update"]
}
# Allow decoding without specifying a decode format
path "transform/decode/us-ssn"
{
    capabilities = ["update"]
}

To demonstrate the functionality, create a token with a policy that will only permit decoding using the last-four decode format.

Define the policy in the file named last-four.hcl.

$ tee last-four.hcl <<EOF
# Allow decoding using only the last-four decode format
path "transform/decode/payments/last-four"
{
    capabilities = ["update"]
}
EOF

$ tee last-four.hcl <<EOF
# Allow decoding using only the last-four decode format
path "transform/decode/payments/last-four"
{
    capabilities = ["update"]
}
EOF

Create the last-four with the policy defined in last-four.hcl.

$ vault policy write last-four last-four.hcl
Success! Uploaded policy: us-ssn-last-four

$ vault policy write last-four last-four.hcl
Success! Uploaded policy: us-ssn-last-four

Create a token with the last-four policy attached and store the token in the variable $LAST_FOUR_TOKEN.

$ LAST_FOUR_TOKEN=$(vault token create -format=json -policy="last-four" | jq -r ".auth.client_token")

$ LAST_FOUR_TOKEN=$(vault token create -format=json -policy="last-four" | jq -r ".auth.client_token")

Using the token, decode values with the payments role, the us-ssn transformation and the last-four decoding format.

$ VAULT_TOKEN=$LAST_FOUR_TOKEN vault write transform/decode/payments/last-four \
     value="766-52-7759" \
     transformation=us-ssn

$ VAULT_TOKEN=$LAST_FOUR_TOKEN vault write transform/decode/payments/last-four \
     value="766-52-7759" \
     transformation=us-ssn

Example output:

Key              Value
---              -----
decoded_value    *** ** 6789

Key              Value
---              -----
decoded_value    *** ** 6789

Trying to use the token to decode with the payments role with the us-ssn transformation with the space-separated decode format, or without specifying a decode format will fail.

$ VAULT_TOKEN=$LAST_FOUR_TOKEN vault write transform/decode/payments \
     value="766-52-7759" \
     transformation=us-ssn

$ VAULT_TOKEN=$LAST_FOUR_TOKEN vault write transform/decode/payments \
     value="766-52-7759" \
     transformation=us-ssn

Example output:

Error writing data to transform/decode/payments: Error making API request.

URL: PUT http://localhost:8200/v1/transform/decode/payments
Code: 403. Errors:

* 1 error occurred:
    * permission denied

Error writing data to transform/decode/payments: Error making API request.

URL: PUT http://localhost:8200/v1/transform/decode/payments
Code: 403. Errors:

* 1 error occurred:
    * permission denied

Define the policy in the file named last-four.hcl.

$ tee last-four.json <<EOF
{
  "policy": "# Allow decoding using only the last-four decode format\npath \"transform/decode/payments/last-four\"\n{\n    capabilities = [\"update\"]\n}\n"
}
EOF

$ tee last-four.json <<EOF
{
  "policy": "# Allow decoding using only the last-four decode format\npath \"transform/decode/payments/last-four\"\n{\n    capabilities = [\"update\"]\n}\n"
}
EOF

Create the last-four with the policy defined in last-four.hcl.

$ curl --header "X-Vault-Token: $VAULT_TOKEN" \
      --request POST \
      --data @last-four.json \
      $VAULT_ADDR/v1/sys/policies/acl/last-four

$ curl --header "X-Vault-Token: $VAULT_TOKEN" \
      --request POST \
      --data @last-four.json \
      $VAULT_ADDR/v1/sys/policies/acl/last-four

Create a token with the last-four policy attached and store the token in the variable $LAST_FOUR_TOKEN.

$ LAST_FOUR_TOKEN=$(curl --request POST --header "X-Vault-Token: $VAULT_TOKEN" \
    --data '{ "policies":"last-four" }' \
    $VAULT_ADDR/v1/auth/token/create | jq -r ".auth.client_token")

$ LAST_FOUR_TOKEN=$(curl --request POST --header "X-Vault-Token: $VAULT_TOKEN" \
    --data '{ "policies":"last-four" }' \
    $VAULT_ADDR/v1/auth/token/create | jq -r ".auth.client_token")

Using the token, decode values with the payments role, the us-ssn transformation and the last-four decoding format.

$ curl --header "X-Vault-Token: $LAST_FOUR_TOKEN" \
      --request POST \
      --data '{"value": '"$ENCODED_OUTPUT"', "transformation": "us-ssn"}' \
      $VAULT_ADDR/v1/transform/decode/payments/last-four | jq ".data"

$ curl --header "X-Vault-Token: $LAST_FOUR_TOKEN" \
      --request POST \
      --data '{"value": '"$ENCODED_OUTPUT"', "transformation": "us-ssn"}' \
      $VAULT_ADDR/v1/transform/decode/payments/last-four | jq ".data"

Example output:

{
  "decoded_value": "*** ** 6789"
}

{
  "decoded_value": "*** ** 6789"
}

Try to use the token to decode with the payments role with the us-ssn transformation with the space-separated decode format, or without specifying a decode format will fail.

$ curl --header "X-Vault-Token: $LAST_FOUR_TOKEN" \
      --request POST \
      --data '{"value": '"$ENCODED_OUTPUT"', "transformation": "us-ssn"}' \
      $VAULT_ADDR/v1/transform/decode/payments

$ curl --header "X-Vault-Token: $LAST_FOUR_TOKEN" \
      --request POST \
      --data '{"value": '"$ENCODED_OUTPUT"', "transformation": "us-ssn"}' \
      $VAULT_ADDR/v1/transform/decode/payments

Example output:

{"errors":["1 error occurred:\n\t* permission denied\n\n"]}

{"errors":["1 error occurred:\n\t* permission denied\n\n"]}

Create custom alphabets

Alphabet defines a set of characters (UTF-8) that is used for FPE to determine the validity of plaintext and ciphertext values.

These are a number of built-in alphabets available to use.

Alphabets	Description
builtin/numeric	Numbers
builtin/alphalower	Lower-case letters
builtin/alphaupper	Upper-case letters
builtin/alphanumericlower	Numbers and lower-case letters
builtin/alphanumericupper	Numbers and upper-case letters
builtin/alphanumeric	Numbers and letters

New alphabets can be created to satisfy the template requirements.

To learn the command, create a non-zero-numeric alphabet which contains non-zero numbers.

Display existing alphabets.

$ vault list transform/alphabet

Keys
----
builtin/alphalower
builtin/alphanumeric
builtin/alphanumericlower
builtin/alphanumericupper
builtin/alphaupper
builtin/numeric

$ vault list transform/alphabet

Keys
----
builtin/alphalower
builtin/alphanumeric
builtin/alphanumericlower
builtin/alphanumericupper
builtin/alphaupper
builtin/numeric

Create an alphabet named non-zero-numeric.

$ vault write transform/alphabet/non-zero-numeric alphabet="123456789"
Success! Data written to: transform/alphabet/non-zero-numeric

$ vault write transform/alphabet/non-zero-numeric alphabet="123456789"
Success! Data written to: transform/alphabet/non-zero-numeric

This new alphabet consists of only characters from the provided set 123456789.

Create an alphabet named non-zero-numeric.

$ curl --header "X-Vault-Token: $VAULT_TOKEN" \
     --request POST \
     --data '{"alphabet": "123456789"}' \
     $VAULT_ADDR/v1/transform/alphabet/non-zero-numeric

$ curl --header "X-Vault-Token: $VAULT_TOKEN" \
     --request POST \
     --data '{"alphabet": "123456789"}' \
     $VAULT_ADDR/v1/transform/alphabet/non-zero-numeric

This new alphabet consists of only characters from the provided set 123456789.

Data masking

Data masking is used to hide sensitive data from those who do not have a clearance to view them. For example, this allows a contractor to test the database environment without having access to the actual sensitive customer information. Data masking has become increasingly important with the enforcement of General Data Protection Regulation (GDPR) introduced in 2018.

The following steps demonstrate the use of masking to obscure your customer's phone number since it is personally identifiable information (PII).

Note

Masking is a unidirectional operation; therefore, encode is the only supported operation.

You will create a phone-number-tmpl template which masks phone numbers with its country code visible.

Create a template named "phone-number-tmpl" with country code.

$ vault write transform/template/phone-number-tmpl type=regex \
    pattern="\+\d{1,2} (\d{3})-(\d{3})-(\d{4})" \
    alphabet=builtin/numeric

$ vault write transform/template/phone-number-tmpl type=regex \
    pattern="\+\d{1,2} (\d{3})-(\d{3})-(\d{4})" \
    alphabet=builtin/numeric

Example output:

Success! Data written to: transform/template/phone-number-tmpl

Success! Data written to: transform/template/phone-number-tmpl

Create a transformation named "phone-number" with the phone-number-tmpl template and allow all roles to use it.

$ vault write transform/transformations/masking/phone-number \
    template=phone-number-tmpl \
    masking_character=# \
    allowed_roles='*'

$ vault write transform/transformations/masking/phone-number \
    template=phone-number-tmpl \
    masking_character=# \
    allowed_roles='*'

Example output:

Success! Data written to: transform/transformations/masking/phone-number

Success! Data written to: transform/transformations/masking/phone-number

The type is set to masking and specifies the masking_character value instead of tweak_source. The default masking character is * if you don't specify one.

Test and verify the newly created phone-number mask transformation by adding the phone-number transformation to the payments role.

$ vault write transform/role/payments \
    transformations=card-number,uk-passport,phone-number

$ vault write transform/role/payments \
    transformations=card-number,uk-passport,phone-number

Example output:

Success! Data written to: transform/role/payments

Success! Data written to: transform/role/payments

Encode a value with the payments role with the phone-number transformation.

$ vault write transform/encode/payments value="+1 123-345-5678" \
    transformation=phone-number

$ vault write transform/encode/payments value="+1 123-345-5678" \
    transformation=phone-number

Example output:

Key              Value
---              -----
encoded_value    +1 ###-###-####

Key              Value
---              -----
encoded_value    +1 ###-###-####

Create a template named "phone-number-tmpl" with country code.

Create an API request payload.

$ tee payload-template.json <<EOF
{
  "type": "regex",
  "pattern": "\\\+\\\d{1,2} (\\\d{3})-(\\\d{3})-(\\\d{4})",
  "alphabet": "builtin/numeric"
}
EOF

$ tee payload-template.json <<EOF
{
  "type": "regex",
  "pattern": "\\\+\\\d{1,2} (\\\d{3})-(\\\d{3})-(\\\d{4})",
  "alphabet": "builtin/numeric"
}
EOF

Create a new template named phone-number-tmpl.

$ curl --header "X-Vault-Token: $VAULT_TOKEN" \
    --request POST \
    --data @payload-template.json \
    $VAULT_ADDR/v1/transform/template/phone-number-tmpl

$ curl --header "X-Vault-Token: $VAULT_TOKEN" \
    --request POST \
    --data @payload-template.json \
    $VAULT_ADDR/v1/transform/template/phone-number-tmpl

Create an API request payload for the transformation named phone-number with the phone-number-tmpl template and allow all roles to use it.

$ tee payload.json <<EOF
{
  "template": "phone-number-tmpl",
  "masking_character": "#",
  "allowed_roles": ["*"]
}
EOF

$ tee payload.json <<EOF
{
  "template": "phone-number-tmpl",
  "masking_character": "#",
  "allowed_roles": ["*"]
}
EOF

Create a transformation named phone-number.

$ curl --header "X-Vault-Token: $VAULT_TOKEN" \
      --request POST \
      --data @payload.json \
      $VAULT_ADDR/v1/transform/transformations/masking/phone-number

$ curl --header "X-Vault-Token: $VAULT_TOKEN" \
      --request POST \
      --data @payload.json \
      $VAULT_ADDR/v1/transform/transformations/masking/phone-number

Verify the newly created phone-number mask transformation by adding the phone-number transformation to the payments role.

$ curl --header "X-Vault-Token: $VAULT_TOKEN" \
    --request POST \
    --data '{"transformations": ["uk-passport","card-number","phone-number"]}' \
    $VAULT_ADDR/v1/transform/role/payments

$ curl --header "X-Vault-Token: $VAULT_TOKEN" \
    --request POST \
    --data '{"transformations": ["uk-passport","card-number","phone-number"]}' \
    $VAULT_ADDR/v1/transform/role/payments

Create an API request payload with a value and the phone-number transformation.

$ tee input.json <<EOF
{
  "value": "+1 123-345-5678",
  "transformation": "phone-number"
}
EOF

$ tee input.json <<EOF
{
  "value": "+1 123-345-5678",
  "transformation": "phone-number"
}
EOF

Encode a value with the payments role with the phone-number transformation.

$ curl --header "X-Vault-Token: $VAULT_TOKEN" \
    --request POST \
    --data @input.json \
    $VAULT_ADDR/v1/transform/encode/payments | jq ".data"

$ curl --header "X-Vault-Token: $VAULT_TOKEN" \
    --request POST \
    --data @input.json \
    $VAULT_ADDR/v1/transform/encode/payments | jq ".data"

Example output:

{
  "encoded_value": "+1 ###-###-####"
}

{
  "encoded_value": "+1 ###-###-####"
}

Note

Refer to the batch input processing section for encoding and decoding multiple secrets in a single API call.

Click the Templates tab and click Create template.
Enter the following information:
- Name - Enter phone-number-tmpl
- Pattern - Enter \+\d{1,2} (\d{3})-(\d{3})-(\d{4})
- Alphabet - Select builtin/numeric
Click Create template to complete.
Click transform in the breadcrumb menu to return to the Templates configuration page.
Click the Transformations tab and click Create transformation.
Enter the following information:
- Name - Enter phone-number
- Type - Select Masking
- Masking Character - Enter #
- Template Choose the phone-number-tmpl you created earlier
- Allowed roles - Enter * to allow all roles to use this transformation
Click Create transformation to complete.

Batch input processing

When you need to encode more than one secret value, you can send multiple secrets in a request payload as batch_input instead of invoking the API endpoint multiple times to encode secrets individually.

Example Scenario 1:

You received a credit card number, British passport number and a phone number of a customer and wish to transform all these secrets using the payments role.

Create an API request payload with multiple values, each with the desired transformation.

$ tee input-multiple.json <<EOF
{
  "batch_input": [
    {
      "value": "1111-1111-1111-1111",
      "transformation": "card-number"
    },
    {
      "value": "123456789",
      "transformation": "uk-passport"
    },
    {
      "value": "+1 123-345-5678",
      "transformation": "phone-number"
    }
  ]
}
EOF

$ tee input-multiple.json <<EOF
{
  "batch_input": [
    {
      "value": "1111-1111-1111-1111",
      "transformation": "card-number"
    },
    {
      "value": "123456789",
      "transformation": "uk-passport"
    },
    {
      "value": "+1 123-345-5678",
      "transformation": "phone-number"
    }
  ]
}
EOF

Encode all the values with the payments role.

$ curl --header "X-Vault-Token: $VAULT_TOKEN" \
     --request POST \
     --data @input-multiple.json \
     $VAULT_ADDR/v1/transform/encode/payments | jq ".data"

$ curl --header "X-Vault-Token: $VAULT_TOKEN" \
     --request POST \
     --data @input-multiple.json \
     $VAULT_ADDR/v1/transform/encode/payments | jq ".data"

Example output:

   {
  "batch_results": [
    {
      "encoded_value": "7998-7227-5261-3751"
    },
    {
      "encoded_value": "908547441"
    },
    {
      "encoded_value": "## ###-###-####"
    }
  ]
}

   {
  "batch_results": [
    {
      "encoded_value": "7998-7227-5261-3751"
    },
    {
      "encoded_value": "908547441"
    },
    {
      "encoded_value": "## ###-###-####"
    }
  ]
}

Example Scenario 2:

An on-premises database stores corporate card numbers and your organization decided to migrate the data to another database. You wish to encode those card numbers before storing them in the new database.

Create a request payload with multiple card numbers.

$ tee payload-batch.json <<EOF
{
  "batch_input": [
    { "value": "1111-1111-1111-1111", "transformation": "card-number" },
    { "value": "2222-2222-2222-2222", "transformation": "card-number" },
    { "value": "3333-3333-3333-3333", "transformation": "card-number" },
    { "value": "4444-4444-4444-4444", "transformation": "card-number" }
  ]
}
EOF

$ tee payload-batch.json <<EOF
{
  "batch_input": [
    { "value": "1111-1111-1111-1111", "transformation": "card-number" },
    { "value": "2222-2222-2222-2222", "transformation": "card-number" },
    { "value": "3333-3333-3333-3333", "transformation": "card-number" },
    { "value": "4444-4444-4444-4444", "transformation": "card-number" }
  ]
}
EOF

Encode all the values with the payments role.

$ BATCH_ENCODED=$(curl --header "X-Vault-Token: $VAULT_TOKEN" \
    --request POST \
    --data @payload-batch.json \
    $VAULT_ADDR/v1/transform/encode/payments | jq ".data") \
    && echo $BATCH_ENCODED

$ BATCH_ENCODED=$(curl --header "X-Vault-Token: $VAULT_TOKEN" \
    --request POST \
    --data @payload-batch.json \
    $VAULT_ADDR/v1/transform/encode/payments | jq ".data") \
    && echo $BATCH_ENCODED

Example output:

{
  "batch_results": [
    {
      "encoded_value": "7998-7227-5261-3751"
    },
    {
      "encoded_value": "2026-7948-2166-0380"
    },
    {
      "encoded_value": "3979-1805-7116-8137"
    },
    {
      "encoded_value": "0196-8166-5765-0438"
    }
  ]
}

{
  "batch_results": [
    {
      "encoded_value": "7998-7227-5261-3751"
    },
    {
      "encoded_value": "2026-7948-2166-0380"
    },
    {
      "encoded_value": "3979-1805-7116-8137"
    },
    {
      "encoded_value": "0196-8166-5765-0438"
    }
  ]
}

Decode the values

Create a request payload with the encoded card numbers.

$ tee payload-batch.json <<EOF
{
  "batch_input": [
    { "value": "$(echo $BATCH_ENCODED | jq -r '.batch_results[0].encoded_value')", "transformation": "card-number" },
    { "value": "$(echo $BATCH_ENCODED | jq -r '.batch_results[1].encoded_value')", "transformation": "card-number" },
    { "value": "$(echo $BATCH_ENCODED | jq -r '.batch_results[2].encoded_value')", "transformation": "card-number" },
    { "value": "$(echo $BATCH_ENCODED | jq -r '.batch_results[3].encoded_value')", "transformation": "card-number" }
  ]
}
EOF

$ tee payload-batch.json <<EOF
{
  "batch_input": [
    { "value": "$(echo $BATCH_ENCODED | jq -r '.batch_results[0].encoded_value')", "transformation": "card-number" },
    { "value": "$(echo $BATCH_ENCODED | jq -r '.batch_results[1].encoded_value')", "transformation": "card-number" },
    { "value": "$(echo $BATCH_ENCODED | jq -r '.batch_results[2].encoded_value')", "transformation": "card-number" },
    { "value": "$(echo $BATCH_ENCODED | jq -r '.batch_results[3].encoded_value')", "transformation": "card-number" }
  ]
}
EOF

Decode all the values with the payments role.

$ curl --header "X-Vault-Token: $VAULT_TOKEN" \
    --request POST \
    --data @payload-batch.json \
    $VAULT_ADDR/v1/transform/decode/payments | jq ".data"

$ curl --header "X-Vault-Token: $VAULT_TOKEN" \
    --request POST \
    --data @payload-batch.json \
    $VAULT_ADDR/v1/transform/decode/payments | jq ".data"

Example output:

{
  "batch_results": [
    {
      "decoded_value": "1111-1111-1111-1111"
    },
    {
      "decoded_value": "2222-2222-2222-2222"
    },
    {
      "decoded_value": "3333-3333-3333-3333"
    },
    {
      "decoded_value": "4444-4444-4444-4444"
    }
  ]
}

{
  "batch_results": [
    {
      "decoded_value": "1111-1111-1111-1111"
    },
    {
      "decoded_value": "2222-2222-2222-2222"
    },
    {
      "decoded_value": "3333-3333-3333-3333"
    },
    {
      "decoded_value": "4444-4444-4444-4444"
    }
  ]
}

Additional discussion

Bring your own key (BYOK)

Vault Enterprise users running version 1.12.0 or greater can use the BYOK functionality to import an existing encryption key generated outside of Vault, and use it with Transform secrets engine.

The target key for import can originate from an HSM or other external source, and must be prepared according to its origin before you can import it.

The example shown here will use a 256-bit AES key, referred to as the target key. To successfully import the target key, you must perform the following operations to prepare it.

Generate an ephemeral 256-bit AES key.
Wrap the target key using the ephemeral AES key with AES-KWP.
Wrap the AES key under the Vault wrapping key using RSAES-OAEP with MGF1 and either SHA-1, SHA-224, SHA-256, SHA-384, or SHA-512.
Delete the ephemeral AES key.
Append the wrapped target key to the wrapped AES key.
Base64 encode the result.

A specific code example for preparing and wrapping the key for import is beyond the scope of this tutorial. For more details about wrapping the key for import including instructions for wrapping key from an HSM, refer to the key wrapping guide.

Before you can wrap the key for import, you must read the wrapping key from Vault so that it can be used to prepare your key.

$ vault read -field=public_key transform/wrapping_key
-----BEGIN PUBLIC KEY-----
MIICIjANBgkqhkiG9w0BAQEFAAOCAg8AMIICCgKCAgEA0aBVXa1Z8X1n1z3smBjU
J+Vm16AznaIrDxWrMAEsQt8xqh33gNMdzsdhNGy0+UzF2mxs9lUbp3uXJxqSaJDB
B+yeSqB4svgcQneg5pZnpZ1Af1s8X6uHTjZZxlHpJ0wumA//zWwg+0TtDij16C5o
Qpep7+UqA9qEcVheCP4puyQXssGH6vwbV4nspNTd1E5RgXe9xpdUKW+ih0pJWbNN
wiW1L6YP8F6dQ4adr6Nm/UUoTpKnvdippT980SLFrkl7v6xh+UKC51bS5fxtQ4Xl
dFCZYWO+Z7HCu3dWEte0Q5vhvbDLCJYKQ/avsQ8EXJtBYzKS7rtvDOYZLOFAgGs4
yeUgVOaIEHXmp+r7CdXnMlo9/5qN02i16HavTgQvaBbPclgH2NWE2JP4sKNsFfyF
DIL9btnUjjgTl6/KaxlVBmZyx96A0V1UTEphziFgeUVw2Rc7PkyOX9lXdAu34A3i
0kgjZC1Y8x/TnGWJ4ofyFBM8RmSaoe4e9C2i2VjHFzJX3ya53Rewy/e+MdvG3tlI
tIEXoanXtTAPbpBJw9tY4FLO8WFsQuBiYg/ovO3ZcYzK5tZwdPnWLjgiXitFchUs
s0lEYFARLap6i/nXxmJnE8jJeAoZ6QSOpNf9ss4rIM9liFvHy+O5Fb3qRLg4zu9M
ERqkZVYfyxp4bChyZwOJD+ECAwEAAQ==
-----END PUBLIC KEY-----

$ vault read -field=public_key transform/wrapping_key
-----BEGIN PUBLIC KEY-----
MIICIjANBgkqhkiG9w0BAQEFAAOCAg8AMIICCgKCAgEA0aBVXa1Z8X1n1z3smBjU
J+Vm16AznaIrDxWrMAEsQt8xqh33gNMdzsdhNGy0+UzF2mxs9lUbp3uXJxqSaJDB
B+yeSqB4svgcQneg5pZnpZ1Af1s8X6uHTjZZxlHpJ0wumA//zWwg+0TtDij16C5o
Qpep7+UqA9qEcVheCP4puyQXssGH6vwbV4nspNTd1E5RgXe9xpdUKW+ih0pJWbNN
wiW1L6YP8F6dQ4adr6Nm/UUoTpKnvdippT980SLFrkl7v6xh+UKC51bS5fxtQ4Xl
dFCZYWO+Z7HCu3dWEte0Q5vhvbDLCJYKQ/avsQ8EXJtBYzKS7rtvDOYZLOFAgGs4
yeUgVOaIEHXmp+r7CdXnMlo9/5qN02i16HavTgQvaBbPclgH2NWE2JP4sKNsFfyF
DIL9btnUjjgTl6/KaxlVBmZyx96A0V1UTEphziFgeUVw2Rc7PkyOX9lXdAu34A3i
0kgjZC1Y8x/TnGWJ4ofyFBM8RmSaoe4e9C2i2VjHFzJX3ya53Rewy/e+MdvG3tlI
tIEXoanXtTAPbpBJw9tY4FLO8WFsQuBiYg/ovO3ZcYzK5tZwdPnWLjgiXitFchUs
s0lEYFARLap6i/nXxmJnE8jJeAoZ6QSOpNf9ss4rIM9liFvHy+O5Fb3qRLg4zu9M
ERqkZVYfyxp4bChyZwOJD+ECAwEAAQ==
-----END PUBLIC KEY-----

The output is the (4096-bit RSA) wrapping key.

$ curl --header "X-Vault-Token: $VAULT_TOKEN" --request GET --silent \
    $VAULT_ADDR/v1/transform/wrapping_key | jq

$ curl --header "X-Vault-Token: $VAULT_TOKEN" --request GET --silent \
    $VAULT_ADDR/v1/transform/wrapping_key | jq

Example output:

{
  "request_id": "03f42f4b-c216-fb54-50f9-a8b2683ba8f6",
  "lease_id": "",
  "renewable": false,
  "lease_duration": 0,
  "data": {
    "public_key": "-----BEGIN PUBLIC KEY-----\nMIICIjANBgkqhkiG9w0BAQEFAAOCAg8AMIICCgKCAgEA0aBVXa1Z8X1n1z3smBjU\nJ+Vm16AznaIrDxWrMAEsQt8xqh33gNMdzsdhNGy0+UzF2mxs9lUbp3uXJxqSaJDB\nB+yeSqB4svgcQneg5pZnpZ1Af1s8X6uHTjZZxlHpJ0wumA//zWwg+0TtDij16C5o\nQpep7+UqA9qEcVheCP4puyQXssGH6vwbV4nspNTd1E5RgXe9xpdUKW+ih0pJWbNN\nwiW1L6YP8F6dQ4adr6Nm/UUoTpKnvdippT980SLFrkl7v6xh+UKC51bS5fxtQ4Xl\ndFCZYWO+Z7HCu3dWEte0Q5vhvbDLCJYKQ/avsQ8EXJtBYzKS7rtvDOYZLOFAgGs4\nyeUgVOaIEHXmp+r7CdXnMlo9/5qN02i16HavTgQvaBbPclgH2NWE2JP4sKNsFfyF\nDIL9btnUjjgTl6/KaxlVBmZyx96A0V1UTEphziFgeUVw2Rc7PkyOX9lXdAu34A3i\n0kgjZC1Y8x/TnGWJ4ofyFBM8RmSaoe4e9C2i2VjHFzJX3ya53Rewy/e+MdvG3tlI\ntIEXoanXtTAPbpBJw9tY4FLO8WFsQuBiYg/ovO3ZcYzK5tZwdPnWLjgiXitFchUs\ns0lEYFARLap6i/nXxmJnE8jJeAoZ6QSOpNf9ss4rIM9liFvHy+O5Fb3qRLg4zu9M\nERqkZVYfyxp4bChyZwOJD+ECAwEAAQ==\n-----END PUBLIC KEY-----\n"
  },
  "wrap_info": null,
  "warnings": null,
  "auth": null
}

{
  "request_id": "03f42f4b-c216-fb54-50f9-a8b2683ba8f6",
  "lease_id": "",
  "renewable": false,
  "lease_duration": 0,
  "data": {
    "public_key": "-----BEGIN PUBLIC KEY-----\nMIICIjANBgkqhkiG9w0BAQEFAAOCAg8AMIICCgKCAgEA0aBVXa1Z8X1n1z3smBjU\nJ+Vm16AznaIrDxWrMAEsQt8xqh33gNMdzsdhNGy0+UzF2mxs9lUbp3uXJxqSaJDB\nB+yeSqB4svgcQneg5pZnpZ1Af1s8X6uHTjZZxlHpJ0wumA//zWwg+0TtDij16C5o\nQpep7+UqA9qEcVheCP4puyQXssGH6vwbV4nspNTd1E5RgXe9xpdUKW+ih0pJWbNN\nwiW1L6YP8F6dQ4adr6Nm/UUoTpKnvdippT980SLFrkl7v6xh+UKC51bS5fxtQ4Xl\ndFCZYWO+Z7HCu3dWEte0Q5vhvbDLCJYKQ/avsQ8EXJtBYzKS7rtvDOYZLOFAgGs4\nyeUgVOaIEHXmp+r7CdXnMlo9/5qN02i16HavTgQvaBbPclgH2NWE2JP4sKNsFfyF\nDIL9btnUjjgTl6/KaxlVBmZyx96A0V1UTEphziFgeUVw2Rc7PkyOX9lXdAu34A3i\n0kgjZC1Y8x/TnGWJ4ofyFBM8RmSaoe4e9C2i2VjHFzJX3ya53Rewy/e+MdvG3tlI\ntIEXoanXtTAPbpBJw9tY4FLO8WFsQuBiYg/ovO3ZcYzK5tZwdPnWLjgiXitFchUs\ns0lEYFARLap6i/nXxmJnE8jJeAoZ6QSOpNf9ss4rIM9liFvHy+O5Fb3qRLg4zu9M\nERqkZVYfyxp4bChyZwOJD+ECAwEAAQ==\n-----END PUBLIC KEY-----\n"
  },
  "wrap_info": null,
  "warnings": null,
  "auth": null
}

The output is a JSON object with a data.public_key field containing the (4096-bit RSA) wrapping key.

Use the wrapping key value at step 3 in the previously detailed preparation steps. Once you have prepared and base64 encoded the ciphertext, export the value to the environment variable IMPORT_CIPHERTEXT.

Example:

$ export IMPORT_CIPHERTEXT=XQiDLXkoN2nE3UXIUnNb4joUSUDyB0OsD/U0T3WpSGuatB2Rd89yj57v1eDb2RDq8tt904iCftDY1c4fJXBlNhvZtRgS45zV+hZZA806YZ2a2ha3rFFEEobRLxs0aCoyq+iqBsOxG4pHG5irtzqfPz5IJRtN0SXxcoOodRm0hGU69++iZO5GkJ8k6Y4513Y/Qwnm/4rBToE9E6toQSTf7exF3yp0wxe/GSfFKCr2aVvYnyK0j+UYNbDeK0M7JOSQ38JmC62oPEfYdI+onyzQxQJsXec/5GOZWsuNnhZtvDNeslnAoC20HiMxl2aMQn4PAvWO+HtlhleXKjaWgDuopgj+qjsSny72pKo2hSntGsamaPqVZQHDmYpIQP4aBsqQeq5bXtDc407cAEPl+MA5huzazdNgMeZqCYZNYFPlUVtZ2dJ6A9ogFE7B8RLAEf4t717kTvlQ0qsXwXOGtoFuw/AQGMTF1CNmCVEJWZBPkxH6GXT6N3QAgwRMYVP6r//IAZ4FKmZ8V4nwmp2uBN0H49XekH1yULa3lWGSXx3CGGy3hYuUHGi6B7mWEH7B0jQgk3EtU1JlgtpTk/jlnL7mpa8VRZefAZqFDX6DoEjioXtqbQAA1Vvkke/PDN3OL6a5aj9kr2UJvlRPfwYsSHw+l8oyazrLs8aKHiaVJAfsVD5y02ipMIGFwaUjswYeW1LxtXg4FVfINzUAoq7z9mvQDby1mvh0Mtr9

$ export IMPORT_CIPHERTEXT=XQiDLXkoN2nE3UXIUnNb4joUSUDyB0OsD/U0T3WpSGuatB2Rd89yj57v1eDb2RDq8tt904iCftDY1c4fJXBlNhvZtRgS45zV+hZZA806YZ2a2ha3rFFEEobRLxs0aCoyq+iqBsOxG4pHG5irtzqfPz5IJRtN0SXxcoOodRm0hGU69++iZO5GkJ8k6Y4513Y/Qwnm/4rBToE9E6toQSTf7exF3yp0wxe/GSfFKCr2aVvYnyK0j+UYNbDeK0M7JOSQ38JmC62oPEfYdI+onyzQxQJsXec/5GOZWsuNnhZtvDNeslnAoC20HiMxl2aMQn4PAvWO+HtlhleXKjaWgDuopgj+qjsSny72pKo2hSntGsamaPqVZQHDmYpIQP4aBsqQeq5bXtDc407cAEPl+MA5huzazdNgMeZqCYZNYFPlUVtZ2dJ6A9ogFE7B8RLAEf4t717kTvlQ0qsXwXOGtoFuw/AQGMTF1CNmCVEJWZBPkxH6GXT6N3QAgwRMYVP6r//IAZ4FKmZ8V4nwmp2uBN0H49XekH1yULa3lWGSXx3CGGy3hYuUHGi6B7mWEH7B0jQgk3EtU1JlgtpTk/jlnL7mpa8VRZefAZqFDX6DoEjioXtqbQAA1Vvkke/PDN3OL6a5aj9kr2UJvlRPfwYsSHw+l8oyazrLs8aKHiaVJAfsVD5y02ipMIGFwaUjswYeW1LxtXg4FVfINzUAoq7z9mvQDby1mvh0Mtr9

Create a new transformation role named physical-access to use for the proximity-card transformation that you will import the key into.

$ vault write transform/role/physical-access transformations=proximity-card
Success! Data written to: transit/role/physical-access

$ vault write transform/role/physical-access transformations=proximity-card
Success! Data written to: transit/role/physical-access

$ curl \
    --header "X-Vault-Token: $VAULT_TOKEN" \
    --request POST \
    --data '{"transformations": ["proximity-card"]}' \
    $VAULT_ADDR/v1/transform/role/physical-access

$ curl \
    --header "X-Vault-Token: $VAULT_TOKEN" \
    --request POST \
    --data '{"transformations": ["proximity-card"]}' \
    $VAULT_ADDR/v1/transform/role/physical-access

Create a new template named identifier to match a proximity card identifier string having the format shown in this example: 8A642EC3-3C8A-40C2-8AC0-A039ECC0FFEE.

$ vault write transform/template/identifier \
    type=regex \
    pattern="(\b[A-Z\d]{8}\b)[-/](\b[A-Z\d]{4}\b)[-/](\b[A-Z\d]{4}\b)[-/](\b[A-Z\d]{4}\b)[-/](\b[A-Z\d]{12}\b)" \
    alphabet=builtin/alphanumericupper

$ vault write transform/template/identifier \
    type=regex \
    pattern="(\b[A-Z\d]{8}\b)[-/](\b[A-Z\d]{4}\b)[-/](\b[A-Z\d]{4}\b)[-/](\b[A-Z\d]{4}\b)[-/](\b[A-Z\d]{12}\b)" \
    alphabet=builtin/alphanumericupper

Example output:

Success! Data written to: transit/template/identifier

Success! Data written to: transit/template/identifier

Create a JSON payload containing the template configuration; because the regular expression used will result in invalid JSON errors unless stringified, you can base64 encode the entire JSON payload, assign it to the environment variable JSON_PAYLOAD_B64, and supply the environment variable for the data in the template creation step.

$ export JSON_PAYLOAD_B64=$(base64 - << EOF
{
  "type": "regex",
  "alphabet": "builtin/alphanumericupper",
  "pattern": "(\b[A-Z\d]{8}\b)[-/](\b[A-Z\d]{4}\b)[-/](\b[A-Z\d]{4}\b)[-/](\b[A-Z\d]{4}\b)[-/](\b[A-Z\d]{12}\b)"
}
EOF
)

$ export JSON_PAYLOAD_B64=$(base64 - << EOF
{
  "type": "regex",
  "alphabet": "builtin/alphanumericupper",
  "pattern": "(\b[A-Z\d]{8}\b)[-/](\b[A-Z\d]{4}\b)[-/](\b[A-Z\d]{4}\b)[-/](\b[A-Z\d]{4}\b)[-/](\b[A-Z\d]{12}\b)"
}
EOF
)

Create the template.

$ curl \
    --silent \
    --header "X-Vault-Token: $VAULT_TOKEN" \
    --request POST \
    --data $JSON_PAYLOAD_B64 \
    $VAULT_ADDR/v1/transform/template/identifier

$ curl \
    --silent \
    --header "X-Vault-Token: $VAULT_TOKEN" \
    --request POST \
    --data $JSON_PAYLOAD_B64 \
    $VAULT_ADDR/v1/transform/template/identifier

Import the key into the proximity-card transformation; add the allowed_roles parameter and specify the physical-access role.

$ vault write transform/transformations/fpe/proximity-card/import \
    ciphertext=$IMPORT_CIPHERTEXT \
    allowed_roles=physical-access \
    template=identifier \
    tweak_source=internal

$ vault write transform/transformations/fpe/proximity-card/import \
    ciphertext=$IMPORT_CIPHERTEXT \
    allowed_roles=physical-access \
    template=identifier \
    tweak_source=internal

Example output:

Success! Data written to: transform/transformations/fpe/proximity-card/import

Success! Data written to: transform/transformations/fpe/proximity-card/import

Create a JSON payload containing ciphertext value and any additional parameters.

$ cat > payload.json << EOF
{
  "ciphertext": "XQiDLXkoN2nE3UXIUnNb4joUSUDyB0OsD/U0T3WpSGuatB2Rd89yj57v1eDb2RDq8tt904iCftDY1c4fJXBlNhvZtRgS45zV+hZZA806YZ2a2ha3rFFEEobRLxs0aCoyq+iqBsOxG4pHG5irtzqfPz5IJRtN0SXxcoOodRm0hGU69++iZO5GkJ8k6Y4513Y/Qwnm/4rBToE9E6toQSTf7exF3yp0wxe/GSfFKCr2aVvYnyK0j+UYNbDeK0M7JOSQ38JmC62oPEfYdI+onyzQxQJsXec/5GOZWsuNnhZtvDNeslnAoC20HiMxl2aMQn4PAvWO+HtlhleXKjaWgDuopgj+qjsSny72pKo2hSntGsamaPqVZQHDmYpIQP4aBsqQeq5bXtDc407cAEPl+MA5huzazdNgMeZqCYZNYFPlUVtZ2dJ6A9ogFE7B8RLAEf4t717kTvlQ0qsXwXOGtoFuw/AQGMTF1CNmCVEJWZBPkxH6GXT6N3QAgwRMYVP6r//IAZ4FKmZ8V4nwmp2uBN0H49XekH1yULa3lWGSXx3CGGy3hYuUHGi6B7mWEH7B0jQgk3EtU1JlgtpTk/jlnL7mpa8VRZefAZqFDX6DoEjioXtqbQAA1Vvkke/PDN3OL6a5aj9kr2UJvlRPfwYsSHw+l8oyazrLs8aKHiaVJAfsVD5y02ipMIGFwaUjswYeW1LxtXg4FVfINzUAoq7z9mvQDby1mvh0Mtr9", "allowed_roles": "physical-access", "template": "identifier", "tweak_source": "internal"
}
EOF

$ cat > payload.json << EOF
{
  "ciphertext": "XQiDLXkoN2nE3UXIUnNb4joUSUDyB0OsD/U0T3WpSGuatB2Rd89yj57v1eDb2RDq8tt904iCftDY1c4fJXBlNhvZtRgS45zV+hZZA806YZ2a2ha3rFFEEobRLxs0aCoyq+iqBsOxG4pHG5irtzqfPz5IJRtN0SXxcoOodRm0hGU69++iZO5GkJ8k6Y4513Y/Qwnm/4rBToE9E6toQSTf7exF3yp0wxe/GSfFKCr2aVvYnyK0j+UYNbDeK0M7JOSQ38JmC62oPEfYdI+onyzQxQJsXec/5GOZWsuNnhZtvDNeslnAoC20HiMxl2aMQn4PAvWO+HtlhleXKjaWgDuopgj+qjsSny72pKo2hSntGsamaPqVZQHDmYpIQP4aBsqQeq5bXtDc407cAEPl+MA5huzazdNgMeZqCYZNYFPlUVtZ2dJ6A9ogFE7B8RLAEf4t717kTvlQ0qsXwXOGtoFuw/AQGMTF1CNmCVEJWZBPkxH6GXT6N3QAgwRMYVP6r//IAZ4FKmZ8V4nwmp2uBN0H49XekH1yULa3lWGSXx3CGGy3hYuUHGi6B7mWEH7B0jQgk3EtU1JlgtpTk/jlnL7mpa8VRZefAZqFDX6DoEjioXtqbQAA1Vvkke/PDN3OL6a5aj9kr2UJvlRPfwYsSHw+l8oyazrLs8aKHiaVJAfsVD5y02ipMIGFwaUjswYeW1LxtXg4FVfINzUAoq7z9mvQDby1mvh0Mtr9", "allowed_roles": "physical-access", "template": "identifier", "tweak_source": "internal"
}
EOF

Import the key.

$ curl \
    --header "X-Vault-Token: $VAULT_TOKEN" \
    --request POST \
    --data @payload.json \
    $VAULT_ADDR/v1/transform/transformations/fpe/proximity-card/import

$ curl \
    --header "X-Vault-Token: $VAULT_TOKEN" \
    --request POST \
    --data @payload.json \
    $VAULT_ADDR/v1/transform/transformations/fpe/proximity-card/import

Try using the newly imported key to encode a value.

$ vault write transform/encode/physical-access value="19742EC3-3C8A-B0C2-23C0-A039ECA9DD23" \
    transformation=proximity-card
Key              Value
---              -----
encoded_value    OGCIWU8A-RBW0-R7FJ-8IZZ-VZZRIWZQ3P5Z

$ vault write transform/encode/physical-access value="19742EC3-3C8A-B0C2-23C0-A039ECA9DD23" \
    transformation=proximity-card
Key              Value
---              -----
encoded_value    OGCIWU8A-RBW0-R7FJ-8IZZ-VZZRIWZQ3P5Z

Create a JSON payload.

$ cat > payload.json << EOF
{
  "value": "19742EC3-3C8A-B0C2-23C0-A039ECA9DD23",
  "transformation": "proximity-card"
}
EOF

$ cat > payload.json << EOF
{
  "value": "19742EC3-3C8A-B0C2-23C0-A039ECA9DD23",
  "transformation": "proximity-card"
}
EOF

Encode the values.

$ curl \
    --silent \
    --header "X-Vault-Token: $VAULT_TOKEN" \
    --request POST \
    --data @payload.json \
    $VAULT_ADDR/v1/transform/encode/physical-access | jq

$ curl \
    --silent \
    --header "X-Vault-Token: $VAULT_TOKEN" \
    --request POST \
    --data @payload.json \
    $VAULT_ADDR/v1/transform/encode/physical-access | jq

Example output:

{
  "request_id": "b7c8af19-2437-4a8f-ea44-c481fba254c6",
  "lease_id": "",
  "renewable": false,
  "lease_duration": 0,
  "data": {
    "encoded_value": "OGCIWU8A-RBW0-R7FJ-8IZZ-VZZRIWZQ3P5Z"
  },
  "wrap_info": null,
  "warnings": null,
  "auth": null
}

{
  "request_id": "b7c8af19-2437-4a8f-ea44-c481fba254c6",
  "lease_id": "",
  "renewable": false,
  "lease_duration": 0,
  "data": {
    "encoded_value": "OGCIWU8A-RBW0-R7FJ-8IZZ-VZZRIWZQ3P5Z"
  },
  "wrap_info": null,
  "warnings": null,
  "auth": null
}

The imported key is working, and the encoded value returned by the proximity-card transformation is using the imported key.

Note

The FPE transformation does not currently support versioning or rotating of its encryption keys.

Next steps

To actually integrate your application with Vault and leverage the transform secrets engine, there are a number of resources which must be configured.

Before the application can even request data transformation, it first needs to authenticate with Vault. Therefore, an auth method (e.g. AWS, Kubernetes, AppRole) must be enabled and configured for the application to use. In addition, an appropriate policy must be created and attached to the client token.

You can codify the Vault configuration using Terraform, and make the configuration repeatable. The Terraform Vault Provider supports the transform secrets engine. It can create policies, enable and configure auth methods and more.

Refer to the Codify Management of Vault Enterprise Using Terraform tutorial to learn how to leverage Terraform.

On the application side, you can run Vault Agent to authenticate with Vault and manage the lifecycle of the client token. Refer to the following tutorials to learn more about Vault Agent:

The Encrypting Data while Preserving Formatting with the Vault Enterprise Transform Secrets Engine blog post introduces some code examples to invoke the transform secrets engine using Vault API.

Clean up

Unset the VAULT_TOKEN environment variable.

$ unset VAULT_TOKEN

$ unset VAULT_TOKEN

Unset the VAULT_ADDR environment variable.
```
$ unset VAULT_ADDR
```
```
$ unset VAULT_ADDR
```

Unset the IMPORT_CIPHERTEXT environment variable.

$ unset IMPORT_CIPHERTEXT

$ unset IMPORT_CIPHERTEXT

Remove the payload files.

$ rm -f input.json payload-batch.json payload.json input-multiple.json last-four.json payload-template.json

$ rm -f input.json payload-batch.json payload.json input-multiple.json last-four.json payload-template.json

You can stop the Vault dev server by pressing Ctrl+C where the server is running. Or, execute the following command.
```
$ pgrep -f vault | xargs kill
```
```
$ pgrep -f vault | xargs kill
```
If you are using Vault Dedicated, you can delete the cluster from the HCP Portal.

Summary

The Transform secrets engine performs secure data transformation and tokenization against the input data. Transformation methods may encompass NIST vetted cryptographic standards such as format-preserving encryption (FPE) via FF3-1, but can also be pseudonymous transformations of the data through other means, such as masking. This tutorial walked through the use of the Transform secrets engine step-by-step.

Limits:

The Transform secrets engine obeys the FF3-1 minimum and maximum sizes on the length of an input, which are a function of the alphabet size.

Help and reference

Next steps

The Encrypting Data with Transform Secrets Engine tutorial introduces demo applications written in Go and Java to learn how to implement the Transform secrets engine API in your application.

If you are interested in data tokenization, refer to the Tokenize Data with Transform Secrets Engine tutorial.

Encrypt Spring app data

Data tokenization

This tutorial also appears in:

22 tutorials

Vault Enterprise
Learn features that are only available to Vault Enterprise.
- Vault