Virtual Event
Join us for the next HashiConf Digital October 12-15, 2020 Register for Free

Security

enterprise

Transform Secrets Engine

»Challenge

Vault's Transit secrets engine provides encryption service; however, the resulting ciphertext does not preserve the original data format or length.

Transit Secrets Engine

Think of a scenario where an organization must cryptographically protect the personally identifiable information (PII) while preserving the data format and length. For example, the database schema expects a certain character length and/or only allow alphanumeric.

The preservation of the original data format or length may be driven by compliance with certain industry standards such as HIPAA or PCI.

»Solution

Vault Enterprise 1.4 with Advanced Data Protection module introduced the Transform secrets engine which handles secure data transformation and tokenization against the provided secrets. Transformation methods encompass NIST vetted cryptographic standards such as format-preserving encryption (FPE) via FF3-1 to encode your secrets while maintaining the data format and length. In addition, it can also be pseudonymous transformations of the data through other means, such as masking.

Transform Secrets Engine

This prevents the need for change in the existing database schema.

»Prerequisites

To perform the tasks described in this guide, you need to have a Vault Enterprise v1.4 or later with Advanced Data Protection module.

NOTE: An interactive tutorial is also available if you do not have a Vault environment to perform the steps described in this guide. Click the Show Tutorial button to launch the tutorial.

»Policy requirements

To perform all tasks demonstrated in this guide, your policy must include the following permissions:

# Work with transform secrets engine
path "transform/*" {
  capabilities = [ "create", "read", "update", "delete", "list" ]
}

# Enable secrets engine
path "sys/mounts/*" {
  capabilities = [ "create", "read", "update", "delete", "list" ]
}

# List enabled secrets engine
path "sys/mounts" {
  capabilities = [ "read", "list" ]
}

If you are not familiar with policies, complete the policies guide.

»Setup the Transform secrets engine

Transform secrets engine configuration workflow:

  1. Enable the transform secrets engine
  2. Create a role containing the transformations that it can perform
  3. Create an alphabet defining a set of characters to use for format-preserving encryption (FPE) if not using the built-in alphabets.
  4. Create a template defining the rules for value matching if not using the built-in template
  5. Create a transformation to specify the nature of the data manipulation

Relationship

Alphabets define a set of valid input/output UTF-8 characters to be used when you perform FPE. In this step, you are going to leverage one of the built-in alphabets. Read the create custom alphabets section to learn how to define your own alphabets.

Data transformation templates are constructed of type (regex), pattern (regex expression) and allowed alphabet used in the input value. Currently, regex is the only supported type. The pattern defines the data format pattern. For example, the most credit card numbers would have a pattern that can be expressed as (\d{4})-(\d{4})-(\d{4})-(\d{4}) in regex.

In this step, the use of the builtin/creditcardnumber template is demonstrated. Read the create custom templates section to learn how to define your own templates.

Transformations define the transformation type (fpe or masking), template, tweak source or the masking character to be used to transform the secrets.

»Tweak source types:

SourceDescription
supplied (default)User provide the tweak source which must be a base64-encoded string
generatedVault generates and returns the tweak source along with the encoded data. The user must securely store the tweak source which will be needed to decrypt the data
internalVault generates a tweak source for the transformation and the same tweak source will be used for every request
  1. Execute the following command to enable the transform secrets engine at transform/.

    $ vault secrets enable transform
    
  2. Create a role named "payments" with "card-number" transformation attached which you will create next.

    $ vault write transform/role/payments transformations=card-number
    
  3. To list existing roles, execute the following command.

    $ vault list transform/role
    Keys
    ----
    payments
    
  4. Create a transformation named "card-number" which will be used to transform credit card numbers. This uses the built-in builtin/creditcardnumber template to perform format-preserving encryption (FPE). The allowed role to use this transformation is payments you just created.

    $ vault write transform/transformation/card-number \
            type=fpe \
            template="builtin/creditcardnumber" \
            tweak_source=internal \
            allowed_roles=payments
    
    Success! Data written to: transform/transformation/card-number
    

    NOTE: The allowed_roles parameter can be set to a wildcard (*) instead of listing role names. Also, the role name can be expressed using globs at the end for pattern matching (e.g. pay*).

    If not using the built-in templates, refer to the Create custom templates section.

  5. To list the existing transformations, execute the following command.

    $ vault list transform/transformation
    Keys
    ----
    card-number
    
  6. To view the details of the newly created card-number transformation, execute the following command.

    $ vault read transform/transformation/card-number
    Key              Value
    ---              -----
    allowed_roles    [payments]
    templates        [builtin/creditcardnumber]
    tweak_source     internal
    type             fpe
    

»Transform secrets

The Vault client applications must have the following in their policy to perform data encoding and decoding using the Transform secrets engine enabled at transform/.

# To request data encoding using any of the roles
# Specify the role name in the path to narrow down the scope
path "transform/encode/*" {
   capabilities = [ "update" ]
}

# To request data decoding using any of the roles
# Specify the role name in the path to narrow down the scope
path "transform/decode/*" {
   capabilities = [ "update" ]
}
  1. To encode secrets using the payments role, execute the following command.

    $ vault write transform/encode/payments value=1111-2222-3333-4444
    
    Key              Value
    ---              -----
    encoded_value    8492-9808-1939-2623
    
  2. To decode the value encoded with payments role, execute the following command.

    $ vault write transform/decode/payments \
            value=8492-9808-1939-2623
    
    Key              Value
    ---              -----
    decoded_value    1111-2222-3333-4444
    

»Create custom templates

Templates define the data format patterns that you wish to keep while transforming the secrets.

To list the existing templates, execute the following command.

$ vault list transform/template

Keys
----
builtin/creditcardnumber
builtin/socialsecuritynumber

»Built-in templates:

TemplateDescription
builtin/creditcardnumberCommonly issued credit card format template
builtin/socialsecuritynumberUS social security number format template

»Example: British Passport

British passport numbers have a pattern of 9 digit numbers.

British Passport

Use regex to define the matching pattern, (\d{9}). The matched values should be within a regex grouping (e.g. (...)). To create a template for British passports, the commands would look as below.

$ vault write transform/template/uk-passport-tmpl \
        type=regex \
        pattern="(\d{9})" \
        alphabet=builtin/numeric

Verification:

Create a transformation named "uk-passport" containing the uk-passport-tmpl template.

$ vault write transform/transformation/uk-passport \
        type=fpe \
        template=uk-passport-tmpl \
        tweak_source=internal \
        allowed_roles=*

Update the payments role to include the uk-passport transformation as well. Alternatively, you can create a new role instead.

$ vault write transform/role/payments transformations=card-number,uk-passport

In this case, you must specify which transformation to use when you send an encode request since the payments role has two transformations associate with it.

$ vault write transform/encode/payments value="123456789" \
        transformation=uk-passport

Key              Value
---              -----
encoded_value    128151714

»Create custom alphabets

Alphabet defines a set of characters (UTF-8) that is used for FPE to determine the validity of plaintext and ciphertext values.

To list existing alphabets, execute the following command.

$ vault list transform/alphabet

Keys
----
builtin/alphalower
builtin/alphanumeric
builtin/alphanumericlower
builtin/alphanumericupper
builtin/alphaupper
builtin/numeric

»Built-in alphabets:

AlphabetsDescription
builtin/numericNumbers
builtin/alphalowerLower-case letters
builtin/alphaupperUpper-case letters
builtin/alphanumericlowerNumbers and lower-case letters
builtin/alphanumericupperNumbers and upper-case letters
builtin/alphanumericNumbers and letters

If the built-in alphabets do not match the allowed set of characters, create a new alphabet to satisfy your need.

$ vault write transform/alphabet/non-zero-numeric alphabet="123456789"

»Data masking

Data masking is used to hide sensitive data from those who do not have a clearance to view them. For example, this allows a contractor to test the database environment without having access to the actual sensitive customer information. Data masking has become increasingly important with the enforcement of General Data Protection Regulation (GDPR) introduced in 2018.

The following steps demonstrate the use of masking to obscure your customer's phone number since it is personally identifiable information (PII).

  1. Create a template named "phone-number-tmpl" with country code.

    $ vault write transform/template/phone-number-tmpl type=regex \
            pattern="(\+\d{1,2}) (\d{3})-(\d{3})-(\d{4})" \
            alphabet=builtin/numeric
    
  2. Create a transformation named "phone-number" containing the phone-number-tmpl template and allow all roles to use it.

    $ vault write transform/transformation/phone-number \
            type=masking \
            template=phone-number-tmpl \
            masking_character=# \
            allowed_roles=*
    

Notice that the type is now set to masking and specifies the masking_character value instead of tweak_source. The default masking character is * if you don't specify one.

  1. Verify the newly created phone-number mask transformation.

    First, add the phone-number transformation to the payments role.

    $ vault write transform/role/payments \
            transformations=card-number,uk-passport,phone-number
    

    Send a test data.

    $ vault write transform/encode/payments value="+1 123-345-5678" \
            transformation=phone-number
    Key              Value
    ---              -----
    encoded_value    ## ###-###-####
    

»Batch input processing

When you need to encode more than one secret value, you can send multiple secrets in a request payload as batch_input instead of invoking the API endpoint multiple times to encode secrets individually.

»Example Scenario 1:

You received a credit card number, British passport number and a phone number of a customer and wish to transform all these secrets using the payments role.

  1. Create an API request payload containing all secrets associated with transformations associated with the payments role.
$ tee input-multiple.json <<EOF
{
  "batch_input": [
    {
      "value": "1111-1111-1111-1111",
      "transformation": "card-number"
    },
    {
      "value": "123456789",
      "transformation": "uk-passport"
    },
    {
      "value": "+1 123-345-5678",
      "transformation": "phone-number"
    }
  ]
}
EOF
  1. Invoke the transform/encode/payments endpoint.
$ curl --header "X-Vault-Token: <TOKEN>" \
       --request POST \
       --data @input-multiple.json \
       http://127.0.0.1:8200/v1/transform/encode/payments | jq ".data"
{
  "batch_results": [
    {
      "encoded_value": "7998-7227-5261-3751"
    },
    {
      "encoded_value": "1RVHG6W7"
    },
    {
      "encoded_value": "## ###-###-####"
    }
  ]
}

»Example Scenario 2:

An on-prem database stores corporate card numbers and your organization decided to migrate the data to another database. You wish to encode those card numbers before storing them in the new database.

  1. To encode multiple card numbers at once, set the values as batch_input. If the role has more than one transformation associate with it, be sure to specify the name of transformation as well.

    Create a request payload containing multiple card numbers.

    $ tee payload-batch.json <<EOF
    {
      "batch_input": [
        { "value": "1111-1111-1111-1111", "transformation": "card-number" },
        { "value": "2222-2222-2222-2222", "transformation": "card-number" },
        { "value": "3333-3333-3333-3333", "transformation": "card-number" },
        { "value": "4444-4444-4444-4444", "transformation": "card-number" }
      ]
    }
    EOF
    

    Execute the transform/encode/payments endpoint.

    $ curl --header "X-Vault-Token: <TOKEN>" \
          --request POST \
          --data @payload-batch.json \
          http://127.0.0.1:8200/v1/transform/encode/payments | jq ".data"
    {
      "batch_results": [
        {
          "encoded_value": "7998-7227-5261-3751"
        },
        {
          "encoded_value": "2026-7948-2166-0380"
        },
        {
          "encoded_value": "3979-1805-7116-8137"
        },
        {
          "encoded_value": "0196-8166-5765-0438"
        }
      ]
    }
    
  2. Similarly, to decode multiple card numbers, set the values as batch_input.

    $ tee payload-batch.json <<EOF
    {
      "batch_input": [
        { "value": "7998-7227-5261-3751", "transformation": "card-number" },
        { "value": "2026-7948-2166-0380", "transformation": "card-number" },
        { "value": "3979-1805-7116-8137", "transformation": "card-number" },
        { "value": "0196-8166-5765-0438", "transformation": "card-number" }
      ]
    }
    EOF
    

    Execute the transform/decode/payments endpoint.

    $ curl --header "X-Vault-Token: <TOKEN>" \
          --request POST \
          --data @payload-batch.json \
          http://127.0.0.1:8200/v1/transform/decode/payments | jq ".data"
    {
      "batch_results": [
        {
          "decoded_value": "1111-1111-1111-1111"
        },
        {
          "decoded_value": "2222-2222-2222-2222"
        },
        {
          "decoded_value": "3333-3333-3333-3333"
        },
        {
          "decoded_value": "4444-4444-4444-4444"
        }
      ]
    }
    

»Summary

The Transform secrets engine performs secure data transformation and tokenization against the input data. Transformation methods may encompass NIST vetted cryptographic standards such as format-preserving encryption (FPE) via FF3-1, but can also be pseudonymous transformations of the data through other means, such as masking. This tutorial walked through the use of the Transform secrets engine step-by-step.

»Help and Reference