NOTE: Transform secrets engine requires Vault Enterprise with the Advanced Data Protection Module.
»Challenge
Vault's Transit secrets engine provides encryption service; however, the resulting ciphertext does not preserve the original data format or length.
Think of a scenario where an organization must cryptographically protect the personally identifiable information (PII) while preserving the data format and length. For example, the database schema expects a certain character length and/or only allow alphanumeric.
The preservation of the original data format or length may be driven by compliance with certain industry standards such as HIPAA or PCI.
»Solution
Vault Enterprise 1.4 with Advanced Data Protection module introduced the Transform secrets engine which handles secure data transformation and tokenization against the provided secrets. Transformation methods encompass NIST vetted cryptographic standards such as format-preserving encryption (FPE) via FF3-1 to encode your secrets while maintaining the data format and length. In addition, it can also be pseudonymous transformations of the data through other means, such as masking.
This prevents the need for change in the existing database schema.
»Prerequisites
To perform the tasks described in this tutorial, you need to have a Vault Enterprise v1.4 or later with Advanced Data Protection module.
NOTE: To explore Vault Enterprise features, you can sign up for a free 30-day trial.
An interactive tutorial is also available if you do not have a Vault environment to perform the steps described in this tutorial. Click the Show Terminal button to start.
»Policy requirements
NOTE: For the purpose of this tutorial, you can use root
token to work
with Vault. However, it is recommended that root tokens are only used for just
enough initial setup or in emergencies. As a best practice, use tokens with
appropriate set of policies based on your role in the organization.
To perform all tasks demonstrated in this tutorial, your policy must include the following permissions:
# Work with transform secrets engine
path "transform/*" {
capabilities = [ "create", "read", "update", "delete", "list" ]
}
# Enable secrets engine
path "sys/mounts/*" {
capabilities = [ "create", "read", "update", "delete", "list" ]
}
# List enabled secrets engine
path "sys/mounts" {
capabilities = [ "read", "list" ]
}
If you are not familiar with policies, complete the policies tutorial.
»Transform secrets engine workflow
Transform secrets engine configuration workflow:
- Enable the
transform
secrets engine - Create a role containing the transformations that it can perform
- Create an alphabet defining a set of characters to use for format-preserving encryption (FPE) if not using the built-in alphabets.
- Create a template defining the rules for value matching if not using the built-in template
- Create a transformation to specify the nature of the data manipulation
Alphabets define a set of valid input/output UTF-8 characters to be used when you perform FPE. In this step, you are going to leverage one of the built-in alphabets. Read the create custom alphabets section to learn how to define your own alphabets.
Data transformation templates are constructed of type (regex
),
pattern (regex
expression) and allowed
alphabet used in the input value. Currently, regex is the only supported
type. The pattern defines the data format pattern. For example, the most credit
card numbers would have a pattern that can be expressed as
(\d{4})-(\d{4})-(\d{4})-(\d{4})
in regex.
In this step, the use of the builtin/creditcardnumber
template is
demonstrated. Read the create custom templates
section to learn how to define your own templates.
Transformations define the transformation template, tweak source or the masking character to be used to transform the secrets.
»Tweak source types:
Source | Description |
---|---|
supplied (default) | User provide the tweak source which must be a base64-encoded 7-digit string |
generated | Vault generates and returns the tweak source along with the encoded data. The user must securely store the tweak source which will be needed to decrypt the data |
internal | Vault generates a tweak source for the transformation and the same tweak source will be used for every request |
NOTE: Tweak source is only applicable to the FPE transformation.
»Create a new transformation
In this section, you are going to:
- Enable the transform secrets engine
- Create a payments role which includes card-number transformation
- Create a card-number transformation which performs format preserving encryption
Execute the following command to enable the
transform
secrets engine attransform/
.$ vault secrets enable transform
Create a role named "payments" with "card-number" transformation attached which you will create next.
$ vault write transform/role/payments transformations=card-number
To list existing roles, execute the following command.
$ vault list transform/role Keys ---- payments
Create a transformation named "card-number" which will be used to transform credit card numbers. This uses the built-in
builtin/creditcardnumber
template to perform format-preserving encryption (FPE). The allowed role to use this transformation ispayments
you just created.$ vault write transform/transformations/fpe/card-number \ template="builtin/creditcardnumber" \ tweak_source=internal \ allowed_roles=payments Success! Data written to: transform/transformations/fpe/card-number
NOTE: The
allowed_roles
parameter can be set to a wildcard (*
) instead of listing role names. Also, the role name can be expressed using globs at the end for pattern matching (e.g.pay*
).You will learn how to define your own template in the Create custom templates section.
To list the existing transformations, execute the following command.
$ vault list transform/transformations/fpe Keys ---- card-number
To view the details of the newly created
card-number
transformation, execute the following command.$ vault read transform/transformations/fpe/card-number Key Value --- ----- allowed_roles [payments] templates [builtin/creditcardnumber] tweak_source internal type fpe
»Transform secrets
The Vault client applications must have the following in their policy to perform
data encoding and decoding using the Transform secrets engine enabled at
transform/
.
# To request data encoding using any of the roles
# Specify the role name in the path to narrow down the scope
path "transform/encode/*" {
capabilities = [ "update" ]
}
# To request data decoding using any of the roles
# Specify the role name in the path to narrow down the scope
path "transform/decode/*" {
capabilities = [ "update" ]
}
Encode a value with the payments
role.
$ vault write transform/encode/payments value=1111-2222-3333-4444
Key Value
--- -----
encoded_value 8492-9808-1939-2623
Decode the value encoded with payments
role.
$ vault write transform/decode/payments \
value=8492-9808-1939-2623
Key Value
--- -----
decoded_value 1111-2222-3333-4444
»Create custom templates
Templates define the data format patterns that you wish to keep while transforming the secrets. In this section, you are going to create a transformation template which encodes British passport numbers.
British passport has a pattern of 9-digit numeric value which can be expressed
as (\d{9})
using the regex. The parentheses tell Vault to encode all values
grouped within; therefore, (\d{9})
will encode the entire passport number.
If you want to encode the last 7 digits leaving the first two numbers
unchanged, the expression should be \d{2}(\d{7})
.
Display all the exiting templates.
$ vault list transform/template
Keys
----
builtin/creditcardnumber
builtin/socialsecuritynumber
Create a template named uk-passport-tmpl
.
$ vault write transform/template/uk-passport-tmpl \
type=regex \
pattern="(\d{9})" \
alphabet=builtin/numeric
This template uses the built-in alphabet, builtin/numeric
.
Create a transformation named uk-passport
with the uk-passport-tmpl
template.
$ vault write transform/transformations/fpe/uk-passport \
template=uk-passport-tmpl \
tweak_source=internal \
allowed_roles=*
Update the payments
role to include the uk-passport
transformation.
$ vault write transform/role/payments transformations=card-number,uk-passport
The payments role has two transformations. Future requests to encode/decode require that the specific transformation is provided.
Encode a value with the payments
role with the uk-passport
.
$ vault write transform/encode/payments value="123456789" \
transformation=uk-passport
Key Value
--- -----
encoded_value 128151714
Remember that you must specify which transformation to use when you send an encode request since the payments role has two transformations associated with it.
»Create custom alphabets
Alphabet defines a set of characters (UTF-8) that is used for FPE to determine the validity of plaintext and ciphertext values.
These are a number of built-in alphabets available to use.
Alphabets | Description |
---|---|
builtin/numeric | Numbers |
builtin/alphalower | Lower-case letters |
builtin/alphaupper | Upper-case letters |
builtin/alphanumericlower | Numbers and lower-case letters |
builtin/alphanumericupper | Numbers and upper-case letters |
builtin/alphanumeric | Numbers and letters |
New alphabets can be created to satisfy the template requirements.
To learn the command, create a non-zero-numeric
alphabet which contains
non-zero numbers.
Display existing alphabets.
$ vault list transform/alphabet
Keys
----
builtin/alphalower
builtin/alphanumeric
builtin/alphanumericlower
builtin/alphanumericupper
builtin/alphaupper
builtin/numeric
Create an alphabet named non-zero-numeric
.
$ vault write transform/alphabet/non-zero-numeric alphabet="123456789"
This new alphabet consists of only characters from the provided set 123456789
.
»Data masking
Data masking is used to hide sensitive data from those who do not have a clearance to view them. For example, this allows a contractor to test the database environment without having access to the actual sensitive customer information. Data masking has become increasingly important with the enforcement of General Data Protection Regulation (GDPR) introduced in 2018.
The following steps demonstrate the use of masking to obscure your customer's phone number since it is personally identifiable information (PII).
NOTE: Masking is a unidirectional operation; therefore, encode
is the
only supported operation.
Now, create a phone-number-tmpl template which masks phone numbers with its country code visible.
Create a template named "phone-number-tmpl" with country code.
$ vault write transform/template/phone-number-tmpl type=regex \
pattern="\+\d{1,2} (\d{3})-(\d{3})-(\d{4})" \
alphabet=builtin/numeric
Create a transformation named "phone-number" with the phone-number-tmpl
template and allow all roles to use it.
$ vault write transform/transformations/masking/phone-number \
template=phone-number-tmpl \
masking_character=# \
allowed_roles=*
The type
is set to masking
and specifies the masking_character
value
instead of tweak_source
. The default masking character is *
if you don't
specify one.
Verify the newly created phone-number
mask transformation.
First, add the phone-number
transformation to the payments
role.
$ vault write transform/role/payments \
transformations=card-number,uk-passport,phone-number
Finally, encode a value with the payments
role with the phone-number
transformation.
$ vault write transform/encode/payments value="+1 123-345-5678" \
transformation=phone-number
Key Value
--- -----
encoded_value +1 ###-###-####
»Batch input processing
When you need to encode more than one secret value, you can send multiple secrets in a request payload as batch_input instead of invoking the API endpoint multiple times to encode secrets individually.
»Example Scenario 1:
You received a credit card number, British passport number and a phone number
of a customer and wish to transform all these secrets using the payments
role.
Create an API request payload with multiple values, each with the desired transformation.
$ tee input-multiple.json <<EOF
{
"batch_input": [
{
"value": "1111-1111-1111-1111",
"transformation": "card-number"
},
{
"value": "123456789",
"transformation": "uk-passport"
},
{
"value": "+1 123-345-5678",
"transformation": "phone-number"
}
]
}
EOF
Encode all the values with the payments
role.
$ curl --header "X-Vault-Token: $VAULT_TOKEN" \
--request POST \
--data @input-multiple.json \
$VAULT_ADDR/v1/transform/encode/payments | jq ".data"
{
"batch_results": [
{
"encoded_value": "7998-7227-5261-3751"
},
{
"encoded_value": "908547441"
},
{
"encoded_value": "## ###-###-####"
}
]
}
»Example Scenario 2:
An on-premise database stores corporate card numbers and your organization decided to migrate the data to another database. You wish to encode those card numbers before storing them in the new database.
Create a request payload with multiple card numbers.
$ tee payload-batch.json <<EOF
{
"batch_input": [
{ "value": "1111-1111-1111-1111", "transformation": "card-number" },
{ "value": "2222-2222-2222-2222", "transformation": "card-number" },
{ "value": "3333-3333-3333-3333", "transformation": "card-number" },
{ "value": "4444-4444-4444-4444", "transformation": "card-number" }
]
}
EOF
Encode all the values with the payments
role.
$ curl --header "X-Vault-Token: $VAULT_TOKEN" \
--request POST \
--data @payload-batch.json \
$VAULT_ADDR/v1/transform/encode/payments | jq ".data"
{
"batch_results": [
{
"encoded_value": "7998-7227-5261-3751"
},
{
"encoded_value": "2026-7948-2166-0380"
},
{
"encoded_value": "3979-1805-7116-8137"
},
{
"encoded_value": "0196-8166-5765-0438"
}
]
}
To decode these values.
First, create a request payload with the encoded card numbers.
$ tee payload-batch.json <<EOF
{
"batch_input": [
{ "value": "7998-7227-5261-3751", "transformation": "card-number" },
{ "value": "2026-7948-2166-0380", "transformation": "card-number" },
{ "value": "3979-1805-7116-8137", "transformation": "card-number" },
{ "value": "0196-8166-5765-0438", "transformation": "card-number" }
]
}
EOF
Decode all the values with the payments
role.
$ curl --header "X-Vault-Token: $VAULT_TOKEN" \
--request POST \
--data @payload-batch.json \
$VAULT_ADDR/v1/transform/decode/payments | jq ".data"
{
"batch_results": [
{
"decoded_value": "1111-1111-1111-1111"
},
{
"decoded_value": "2222-2222-2222-2222"
},
{
"decoded_value": "3333-3333-3333-3333"
},
{
"decoded_value": "4444-4444-4444-4444"
}
]
}
»Next steps
To actually integrate your application with Vault and leverage the transform secrets engine, there are a number of resources must be configured.
Before the application can even request data transformation, it first needs to authenticate with Vault. Therefore, an auth method (e.g. AWS, Kubernetes, AppRole) must be enabled and configured for the application to use. In addition, an appropriate policy must be created and attached to the client token.
You can codify the Vault configuration using Terraform, and make the configuration repeatable. Terraform Vault Provider supports transform secrets engine. It can create policies, enable and configure auth methods and more.
Refer to the Codify Management of Vault Enterprise tutorial to learn how to leverage Terraform.
On the application side, you can run Vault Agent to authenticate with Vault and manage the lifecycle of the client token. Refer to the following tutorials to learn more about Vault Agent:
The Encrypting Data while Preserving Formatting with the Vault Enterprise Transform Secrets Engine blog post introduces some code examples to invoke the transform secrets engine using Vault API.
»Summary
The Transform secrets engine performs secure data transformation and tokenization against the input data. Transformation methods may encompass NIST vetted cryptographic standards such as format-preserving encryption (FPE) via FF3-1, but can also be pseudonymous transformations of the data through other means, such as masking. This tutorial walked through the use of the Transform secrets engine step-by-step.
»Limits:
The Transform secret engine obeys the FF3-1 minimum and maximum sizes on the length of an input, which are a function of the alphabet size.
»Help and Reference
- Transform Secrets Engine (API)
- Transform Secrets Engine
- Encrypting Data while Preserving Formatting with the Vault Enterprise Transform Secrets Engine
New: Vault 1.6.0 introduced tokenization transformation. Refer to the Tokenize Data with Transform Secrets Engine tutorial to learn more.