Virtual Event
Join us for the next HashiConf Digital October 12-15, 2020 Register for Free

Vault 1.5 Release Highlights

Protecting Vault with Resource Quotas

Vault is an API driven system that all communication between the clients and Vault are done through Vault API.

As the number of client applications increases, rogue applications can deprive Vault performance. The issue is often caused by applications that are:

  • Generating an unbounded number of leases and/or loading too much data into Vault leading to exhausting Consul storage backend's available memory
  • Consuming an erroneously large amount of bandwidth leading to internal throttling of the Vault cluster as a whole

»Solution

Vault 1.5 introduced resource quotas to protect your Vault environment's stability and network, as well as storage resource consumption from runaway application behavior and distributed denial of service (DDoS) attack.

The Vault operators can control how applications request resources from Vault, and Vault's storage and network infrastructure by setting the following:

  • Rate Limit Quotas: Limit maximum amount of requests per second (RPS) to a system or mount to protect network bandwidth
  • Lease Count Quotas (Vault Enterprise only): Cap number of leases generated in a system or mount to protect system stability and storage performance at scale

»Prerequisites

To perform the tasks described in this guide, you need Vault v1.5 or later. Refer to the Getting Started guide to install Vault. Make sure that your Vault server has been initialized and unsealed.

NOTE: If you do not have a Vault environment to perform the steps described in this guide. Click the Show Tutorial button to launch a tutorial in Katacoda.

»Configure rate limit resource quotas

To configure the resource quota, use the sys/quotas/config endpoint.

ParameterTypeDescription
enable_rate_limit_audit_loggingbooleanEnable or disable audit logging when requests get rejected due to rate limit quota violations. By default, audit logging is disabled (false).

By default, the requests rejected due to rate limit quota violations are not written to the audit log. Therefore, if you wish to log the rejected requests for traceability, you must set the enable_rate_limit_audit_logging to true. The requests rejected due to reaching the lease count quotas are always logged that you do not need to set any parameter.

  1. Enable file audit device.

    $ vault audit enable file file_path="/var/log/vault-audit.log"
    

    You can set the target file_path to your desired location.

  2. To enable the audit logging for rate limit quotas, execute the following command.

    $ vault write sys/quotas/config enable_rate_limit_audit_logging=true
    
  3. Read the quota configuration to verify.

    $ vault read sys/quotas/config
    
    Key                                Value
    ---                                -----
    enable_rate_limit_audit_logging    true
    

»Manage resource quotas

To set the rate limit quotas and lease count quotas, use the sys/quotas/<type> endpoint. Each resource quota has a name to identify the quota rule. To manage an individual quota rule, the endpoint becomes sys/quotas/<type>/<name>.

The <type> can be:

»Set rate limit quotas

Rate limit quotas are designed to protect Vault against external distributed denial of service (DDoS) attacks and are fundamental to Vault's security model. Therefore, it is a part of Vault's core feature set available in both OSS and Enterprise.

To set rate limit quotas, use the sys/quotas/rate-limit/<name> endpoint.

»Parameters

ParameterTypeDescription
namestringName of the quota rule
pathstringPath to the mount or namespace to apply the quota rule. A blank path configures a global rate limit quota
ratefloatRate for the number of allowed requests per second (RPS)
  1. Create a rate limit quota named, "global-rate" which limits inbound workload to 30,000 requests per minute. To calculate the rate (request per second), 30,000 divided by 60 seconds (1 minute) which is 500.

    $ vault write sys/quotas/rate-limit/global-rate rate=500
    
  2. Read the global-rate rule to verify its configuration.

    $ vault read sys/quotas/rate-limit/global-rate
    
    Key      Value
    ---      -----
    name     global-rate
    path     n/a
    rate     500
    type     rate-limit
    

»Vault Enterprise Example

Consider that you have K/V v2 secrets engine enabled at kv-v2 under us-west namespace. Create a rate limit quota named, "orders" which limits the incoming requests against kv-v2 to 1,000 requests per minute maximum. In this case, the RPS becomes 16.67 which is 1,000 divided by 60 seconds (1 minute).

  1. Create a namespace, "us-west".

    $ vault namespace create us-west
    
  2. Enable kv-v2 secrets engine in the us-west namespace.

    $ vault secrets enable -ns="us-west" kv-v2
    
  3. Now, create the orders quota rule.

    $ vault write sys/quotas/rate-limit/orders \
         path="us-west/kv-v2" \
         rate=16.67
    

»Set lease count quotas

Lease count quota is designed to protect Vault from a large volume of leases and tokens persisted in Vault for a longer period of time than necessary and pressuring its storage backend. Large scale Vault Enterprise deployments have this governance challenge and organizational complexity ensuring there are guard rails for users in Vault as a service environment to not jeopardize system stability.

»Parameters

ParameterTypeDescription
namestringName of the quota rule
pathstringPath to the mount or namespace to apply the quota rule. A blank path configures a global lease count quota
max_leasesintMaximum number of leases allowed by the quota rule

Assuming that you have a database secrets engine enabled at "postgres" in the us-west namespace.

$ vault secrets enable -ns=us-west -path=postgres database

Create a lease count quota named, "db-creds" which limits the incoming requests for a new set of DB credentials to 100 requests per second maximum.

$ vault write sys/quotas/lease-count/db-creds max_leases=100 \
    path="us-west/postgres"

»Test to understand the resource quotas

Now that you learned the basic commands, let's test to see how it works.

»Rate limit quota test

  1. Enable kv-v2 secrets engine at test.

    $ vault secrets enable -path=test kv-v2
    
  2. Create a rate limit quota, "rate-test" with very low settings so that you can see the output when the quota rule is exceeded. Set the rate to 2 against the test path.

    $ vault write sys/quotas/rate-limit/rate-test path=test rate=1
    
  3. Create a shortcut script, rate-limit-test.sh which invokes the test path.

    $ tee rate-limit-test.sh <<EOF
    vault kv put test/creds1 pasword="test-password-1"
    vault kv put test/creds2 pasword="test-password-2"
    vault kv put test/creds3 pasword="test-password-3"
    EOF
    
  4. Ensure that the script is executable.

    $ chmod +x rate-limit-test.sh
    
  5. Run the script to see how the quota rule behaves.

    $ ./rate-limit-test.sh
    

    The first command completes successfully; however, you should see the following message for the rest.

    Error writing data to test/data/creds3: Error making API request.
    
    URL: PUT http://127.0.0.1:8200/v1/test/data/creds3
    Code: 429. Errors:
    
     request path "test/data/creds3": rate limit quota exceeded
    
  6. Earlier, you configured the resource quotas to enable audit logging of requests that were rejected due to rate limit quota rule violation. Inspect your audit log for its entry.

    $ more /var/log/vault-audit.log | jq
    

    If your audit log path is not /var/log/vault-audit.log, be sure to set it to the correct path.

    ...snip...
      "request": {
        "id": "ec0dbe14-9389-d383-361e-a6765f89b256",
        "operation": "update",
        "namespace": {
          "id": "root"
        },
        "path": "test/data/creds3",
        "data": {
          "data": {
            "pasword": "test-password-3"
          },
          "options": {}
        },
        "remote_address": "127.0.0.1"
      },
      "error": "request path \"test/data/creds3\": rate limit quota exceeded"
    }
    

    You should find an error message indicating that rate limit quota was exceeded that test-password-3 failed to be written at test/data/creds3. You can trace the audit log to see how many requests were rejected due to the rate limit quota. It may be working as expected or you may find suspicious activities against a specific path.

  7. When you are done exploring, delete the rate-test quota.

    $ vault delete sys/quotas/rate-limit/rate-test
    
    Success! Data deleted (if it existed) at: sys/quotas/rate-limit/rate-test
    

»Lease count quota test

  1. Similarly, create a very limited lease count quota named, "lease-test" which applies on the root level. It only allows 3 tokens and leases to be stored.

    $ vault write sys/quotas/lease-count/lease-test max_leases=3
    

    To create a root level quota rule, simply do not set the target path.

  2. Create a shortcut script, lease-count-test.sh which invokes the test path.

    $ tee lease-count-test.sh <<EOF
    vault token create -policy=default
    vault token create -policy=default
    vault token create -policy=default
    vault token create -policy=default
    EOF
    
  3. Make sure that the script is executable.

    $ chmod +x lease-count-test.sh
    
  4. Run the script to see how the quota rule behaves.

    $ ./lease-count-test.sh
    

    Three tokens were created successfully; however, the fourth request failed due to the lease count quota. Your output should look similar to follow.

    Key                  Value
    ---                  -----
    token                s.du7ek5IdHNuwHCl5eGVpcGU2
    token_accessor       1CAK3voVgga5J62Q47NwwNMM
    token_duration       768h
    token_renewable      true
    token_policies       ["default"]
    identity_policies    []
    policies             ["default"]
    Key                  Value
    ---                  -----
    token                s.lIINYrHe3dDfzHy4ZZ0SuOtN
    token_accessor       x6Xv6qagjUhvVYWnVq79VFVz
    token_duration       768h
    token_renewable      true
    token_policies       ["default"]
    identity_policies    []
    policies             ["default"]
    Key                  Value
    ---                  -----
    token                s.MdcsHZEsbhogH4zIIcjtLYJk
    token_accessor       1Gs5NvBTMYNfJyIMc3YBtVbV
    token_duration       768h
    token_renewable      true
    token_policies       ["default"]
    identity_policies    []
    policies             ["default"]
    Error creating token: Error making API request.
    
    URL: POST http://127.0.0.1:8200/v1/auth/token/create
    Code: 429. Errors:
    
     1 error occurred:
         request path "auth/token/create": lease count quota exceeded
    

    If your Vault server already has client tokens, the lease count quota may exceed sooner.

    Also, you can find the lease count quota exceeded error in the audit log. (Be sure to set it to the correct audit log path for your environment.)

    $ tail -f /var/log/vault-audit.log | jq
    
    ...snip...
      "response": {
        "mount_type": "token"
      },
      "error": "1 error occurred:\n\t* request path \"auth/token/create\": lease count quota exceeded\n\n"
    }
    
  5. If you revoke one of the tokens, you should be able to request a new one.

    Example:

    $ vault token revoke s.MdcsHZEsbhogH4zIIcjtLYJk
    
    Success! Revoked token (if it existed)
    

    Now, request a new one.

    $ vault token create -policy=default
    

    The best practice is to set the tokens and leases' time-to-live (TTL) to be short and don't let them hang around longer than necessary. The lease count quotas allow you to set the upper limit to protect your Vault environment from running into an issue due to a lack of token and lease governance.

  6. When you are done exploring, delete the lease-test quota rule.

    $ vault delete sys/quotas/lease-count/lease-test
    

»Next steps

In this tutorial, you learned the basic commands to set resource quotas to protect your Vault environment. To leverage this feature, you need Vault 1.5 or later.

Rate limit quotas allow Vault operators to set inbound request rate limits which can be set on the root level or a specific path. This is available in both Vault OSS and Vault Enterprise.

Lease count quotas require Vault Enterprise Platform and allow operators to set the maximum number of tokens and leases to be persisted at any given time. This can prevent Vault from exhausting the resource on the storage backend.

You also learned that audit logging can be enabled to trace the number of requests that were rejected due to the rate limit quota.

»Help and Reference