Security

enterprise

Secure Multi-Tenancy with Namespaces

Everything in Vault is path-based, and often uses the terms path and namespace interchangeably. The application namespace pattern is a useful construct for providing Vault as a service to internal customers, giving them the ability to implement secure multi-tenancy within Vault in order to provide isolation and ensure teams can self-manage their own environments.

Personas

The scenario described in this guide introduces the following personas:

  • operations is the cluster-level administrator with privileged policies
  • org-admin is the organization-level administrator
  • team-admin is the team-level administrator

Challenge

When Vault is primarily used as a central location to manage secrets, multiple organizations within a company may need to be able to manage their secrets in a self-serving manner. This means that a company needs to implement a Vault as a Service model allowing each organization (tenant) to manage their own secrets and policies. The most importantly, tenants should be restricted to work only within their tenant scope.

Multi-Tenant

Solution

Create a namespace dedicated to each team, organization, or app where they can perform all necessary tasks within their tenant namespace.

Each namespace can have its own:

  • Policies
  • Auth Methods
  • Secrets Engines
  • Tokens
  • Identity entities and groups

Prerequisites

To perform the tasks described in this guide, you need to have a Vault Enterprise environment.

Scenario Introduction

In this guide, you are going to create a namespace dedicated to the Education organization which has Training and Certification teams. Delegate operational tasks to the team admins so that the Vault cluster operators won't have to be involved.

Scenario

he following steps are demonstrated:

  1. Create namespaces
  2. Write policies
  3. Setup entities and groups
  4. Test the organization admin user
  5. Test the team admin user
  6. Audit ambient credentials

Step 1: Create namespaces

(Persona: operations)

To create a new namespace, run: vault namespace create <namespace_name>

  1. Create a namespace dedicated to the education organizations.

    $ vault namespace create education
    
  2. Create child namespaces called training under the education namespace.

    $ vault namespace create -namespace=education training
    
  3. Create child namespaces called certification under the education namespace.

    $ vault namespace create -namespace=education certification
    
  4. List the existing namespaces on the root.

    $ vault namespace list
    education/
    
  5. List the existing namespace on the education namespace.

    $ vault namespace list -namespace=education
    certification/
    training/
    

Step 2: Write Policies

(Persona: operations)

In this scenario, there is an organization-level administrator who is a superuser within the scope of the education namespace. Also, there is a team-level administrator for training and certification.

Policy for education admin

Requirements:

  • Create and manage namespaces
  • Create and manage policies
  • Enable and manage secrets engines
  • Create and manage entities and groups
  • Manage tokens

edu-admin.hcl

# Manage namespaces
path "sys/namespaces/*" {
   capabilities = ["create", "read", "update", "delete", "list", "sudo"]
}

# Manage policies
path "sys/policies/acl/*" {
   capabilities = ["create", "read", "update", "delete", "list", "sudo"]
}

# List policies
path "sys/policies/acl" {
   capabilities = ["list"]
}

# Enable and manage secrets engines
path "sys/mounts/*" {
   capabilities = ["create", "read", "update", "delete", "list"]
}

# List available secrets engines
path "sys/mounts" {
  capabilities = [ "read" ]
}

# Create and manage entities and groups
path "identity/*" {
   capabilities = ["create", "read", "update", "delete", "list"]
}

# Manage tokens
path "auth/token/*" {
   capabilities = ["create", "read", "update", "delete", "list", "sudo"]
}

Policy for training admin

Requirements:

  • Create and manage child-namespaces
  • Create and manage policies
  • Enable and manage secrets engines

training-admin.hcl

# Manage namespaces
path "sys/namespaces/*" {
   capabilities = ["create", "read", "update", "delete", "list", "sudo"]
}

# Manage policies
path "sys/policies/acl/*" {
   capabilities = ["create", "read", "update", "delete", "list", "sudo"]
}

# List policies
path "sys/policies/acl" {
  capabilities = ["list"]
}

# Enable and manage secrets engines
path "sys/mounts/*" {
   capabilities = ["create", "read", "update", "delete", "list"]
}

# List available secrets engines
path "sys/mounts" {
  capabilities = [ "read" ]
}

Now, let's deploy the policies!

To target a specific namespace, you can do one of the following:

Option 1: Set VAULT_NAMESPACE so that all subsequent CLI commands will be executed against that particular namespace.

$ export VAULT_NAMESPACE=<namespace_name>
$ vault policy write <policy_name> <policy_file>

Option 2: Specify the target namespace with -namespace flag

$ vault policy write -namespace=<namespace_name> <policy_name> <policy_file>

Since you have to deploy policies onto "education" and "education/training" namespaces, use "-namespace" flag instead of environment variable.

Create edu-admin policy under 'education' namespace.

$ vault policy write -namespace=education edu-admin edu-admin.hcl

Create training-admin policy under education/training namespace.

$ vault policy write -namespace=education/training training-admin training-admin.hcl

Step 3: Setup entities and groups

(Persona: operations)

In the education namespace, create an entity, Bob Smith with edu-admin policy attached. Add a userpass user, bob as an entity alias. By default, Bob Smith has no visibility into the education/training namespace since the bob user was defined in the education namespace.

You are going to create an internal group named, Training Admin in the education/training namespace with training-admin policy attached. To grant the training-admin policy for bob, add the Bob Smith entity to the Training Admin group as a member entity.

Entities and Groups

  1. Enable the userpass auth method.

    $ vault auth enable -namespace=education userpass
    
  2. Create a new user, bob with password, "training".

    $ vault write auth/userpass/users/bob password="training"
    
  3. Create a user bob under the education namespace.

    $ vault write -namespace=education \
            auth/userpass/users/bob password="training"
    
  4. Create an entity for Bob Smith with edu-adminpolicy attached. Save the generated entity ID in a file namedentity_id.txt`.

    $ vault write -namespace=education -format=json identity/entity name="Bob Smith" \
            policies="edu-admin" | jq -r ".data.id" > entity_id.txt
    
  5. Get the mount accessor for userpass auth method and save it in accessor.txt.

    $ vault auth list -namespace=education -format=json \
            | jq -r '.["userpass/"].accessor' > accessor.txt
    
  6. Create an entity alias for Bob Smith to attach bob.

    $ vault write -namespace=education identity/entity-alias name="bob" \
            canonical_id=$(cat entity_id.txt) mount_accessor=$(cat accessor.txt)
    
  7. Create a group, "Training Admin" in education/training namespace with Bob Smith entity as its member.

    $ vault write -namespace=education/training identity/group \
            name="Training Admin" policies="training-admin" \
            member_entity_ids=$(cat entity_id.txt)
    

Step 4: Test the Bob Smith entity

(Persona: org-admin)

Log in as bob into the education namespace.

$ vault login -namespace=education -method=userpass \
        username="bob" password="training"

Key                    Value
---                    -----
token                  s.zLNbFJQFWaMR7R5tgobn6xHg.1Vi61
token_accessor         xcCNEsRStvQcPEp4AKrkWikk.1Vi61
token_duration         768h
token_renewable        true
token_policies         ["default"]
identity_policies      ["edu-admin"]
policies               ["default" "edu-admin"]
token_meta_username    bob

Notice that the generated token contains the namespace ID which was created in (e.g. s.zLNbFJQFWaMR7R5tgobn6xHg.1Vi61). User bob only has default policy attached to his token (token_policies); however, he inherited the edu-admin policy from the Bob Smith entity (identity_policies).

Test to make sure that bob can create a namespace, enable secrets engine, and whatever else that you want to verify against the education namespace.

$ export VAULT_NAMESPACE="education"

Verify that you can create a new namespace called web-app.

$ vault namespace create web-app
Success! Namespace created at: education/web-app/

Verify that you can enable key/value v2 secrets engine at edu-secret.

$ vault secrets enable -path=edu-secret kv-v2
Success! Enabled the kv-v2 secrets engine at: edu-secret/

Optionally, you can create new policies to test that bob can perform the operations as expected. When you are done testing, unset the VAULT_NAMESPACE environment variable.

$ unset VAULT_NAMESPACE

Step 5: Test the Training Admin group

(Persona: team-admin)

Stay logged in as bob and look up the token details.

$ vault token lookup

Key                            Value
---                            -----
  # ...snip...
external_namespace_policies    map[ygcTv:[training-admin]]
id                             s.zLNbFJQFWaMR7R5tgobn6xHg.1Vi61
identity_policies              [edu-admin]
issue_time                     2019-09-26T16:29:33.960232-07:00
meta                           map[username:bob]
namespace_path                 education/
num_uses                       0
orphan                         true
path                           auth/userpass/login/bob
policies                       [default]
renewable                      true
ttl                            767h53m55s
type                           service

Verify that bob can perform the operations permitted by the training-admin policy.

Set the target namespace as an env variable.

$ export VAULT_NAMESPACE="education/training"

Create a new namespace called vault-training.

$ vault namespace create vault-training

Key     Value
---     -----
id      nQjeG
path    education/training/vault-training/

Enable key/value v1 secrets engine at team-secret.

$ vault secrets enable -path=team-secret -version=1 kv
Success! Enabled the kv secrets engine at: team-secret/

When you are done testing, unset the VAULT_NAMESPACE environment variable.

$ unset VAULT_NAMESPACE

Summary:

As this guide demonstrated, each namespace you created behaves as an isolated Vault environment. By default, there is no visibility into other namespaces regardless of its hierarchical relationship. In order for Bob to operate in education/training namespace, you can enable an auth method in the education/training namespace so that he can log in. Or, as demonstrated in this guide, you can use Vault identity to associate entities and groups defined in different namespaces.

Step 6: Audit ambient credentials

(Persona: operator)

Many auth and secrets providers, such as AWS, Azure, GCP, and AliCloud, use ambient credentials to authenticate API calls. For example, AWS may:

  1. Use an access key and secret key configured in Vault.
  2. If not present, check for environment variables such as "AWS_ACCESS_KEY_ID" and "AWS_SECRET_ACCESS_KEY".
  3. If not present, load credentials configured in "~/.aws/credentials".
  4. If not present, use instance metadata.

This becomes a problem if these ambient credentials are not intended to be used within a particular namespace.

For example, suppose that your Vault server is running on an AWS EC2 instance. You give the owner of a namespace a particular set of permissions to use for AWS. However, that owner does not configure them. So, Vault falls back to using the credentials available in instance metadata, leading to a privilege escalation.

To handle this:

  • Ensure no environment variables are available that could grant a privilege escalation.
  • Ensure that any privileges granted through instance metadata (in this example) or other ambient identity info represent a loss of privilege.
  • Directly configure the correct credentials in namespaces, and restrict access to that endpoint so credentials can't later be edited to use ambient credentials.

Additional Discussion

Leveraging Identity for Auth Methods with External Groups

For simplicity, this guide used the username and password (userpass) auth method which was enabled in the education namespace as well as the education/training namespace.

However, most likely, your organization uses an auth method such as ldap or okta and map appropriate policies to those externally defined groups. For these types of auth methods, you have two options:

  1. Enable the auth method in each namespace the same way Step 3 demonstrated with userpass.
  2. Enable the auth method in the root namespace and use Identity Groups to pull in external groups and map policies in each namespace.

The second option enables an auth method in the root namespace rather than enable in multiple namespaces. The following steps demonstrate the second option to create the "Training Admin" group as described in this guide.

First, enable and configure the desired auth method (e.g. LDAP) in the root namespace.

$ vault auth enable ldap

Configure the LDAP auth method.

$ vault write auth/ldap/config \
        url="ldap://ldap.example.com" \
        userdn="ou=Users,dc=example,dc=com" \
        groupdn="ou=Groups,dc=example,dc=com" \
        groupfilter="(&(objectClass=group)(member:1.2.840.113556.1.4.1941:={{.UserDN}}))" \
        groupattr="cn" \
        upndomain="example.com" \
        certificate=@ldap_ca_cert.pem \
        insecure_tls=false \
        starttls=true

For each LDAP group to which you want to map policies, create an external group in the root namespace with an alias whose name exactly matches the LDAP group name.

Get the mount accessor for ldap auth method and save it in accessor.txt.

$ vault auth list -format=json \
        | jq -r '.["ldap/"].accessor' > accessor.txt

Create an external group and save the generated group ID in group_id.txt.

$ vault write -format=json identity/group name="training_admin_root" \
        type="external" \
        | jq -r ".data.id" > group_id.txt

Create a group alias whose name exactly matches the LDAP group name. The following example uses a group name in LDAP is ops_training.

$ vault write -format=json identity/group-alias name="ops_training" \
        mount_accessor=$(cat accessor.txt) \
        canonical_id=$(cat group_id.txt)

In the education/training namespace, create an internal group which has the external group (training_admin_root) as its member. Attach the training-admin policy to this internal group. Remember that you created this policy inside of the education/training namespace in Step 2.

$ vault write -namespace=education/training identity/group \
        name="Training Admin" \
        policies="training-admin" \
        member_group_ids=$(cat group_id.txt)

Let's assume the user, "bob_smith" belongs to the LDAP ops_training group.

$ vault login -method=ldap username=bob_smith
Password (will be hidden):

Success! You are now authenticated.
# ...snip...

When you check the returned token's properties, the output should display training-admin as its external_namespace_policies.

$ vault token Lookup

Key                            Value
---                            -----
...snip...
external_namespace_policies    map[9dKXw:[training-admin]]
id                             s.10LCp6O5xnweGVm1eyJbp127
identity_policies              <nil>
...snip...
policies                       [default]
renewable                      true
ttl                            767h59m11s
type                           service

Since you did not assign any policy when you created the training_admin_root external group, the identity_policies shows nil. Therefore, bob_smith's token only has the default policy in the root namespace. However, it has training-admin policy in the external namespace of ID, 9dKXw (in this example, it is the ID of education/training namespace).

Set the target namespace as an environment variable for convenience, and bob_smith is ready to operate against the education/training namespace.

$ export VAULT_NAMESPACE="education/training"

The namespaces feature is designed to create an isolation around each namespace so that Vault tenants can operate independently by having a mini-Vault environment of their own. However, the second option leaves some dependency on the root namespace. The root-level admin is responsible for creating the mapping between the LDAP groups and the Vault's identity groups. The users log into the root namespace instead of logging in the target namespace.

Policy with namespaces

In this guide, you created policies in each namespace (education and education/training). Therefore, you did not have to specify the target namespace in the policy paths.

If you want to create policies in the root namespace to control education and education/training namespaces, prepend the namespace in the paths.

For example:

# Manage policies in the 'education' namespace
path "education/sys/policies/*" {
   capabilities = ["create", "read", "update", "delete", "list", "sudo"]
}

# Manage tokens in the 'education' namespace
path "education/auth/token/*" {
   capabilities = ["create", "read", "update", "delete", "list", "sudo"]
}

# Manage policies under 'education/training' namespace
path "education/training/sys/policies/*" {
   capabilities = ["create", "read", "update", "delete", "list", "sudo"]
}

# Manage tokens in the 'education/training' namespace
path "education/training/auth/token/*" {
   capabilities = ["create", "read", "update", "delete", "list", "sudo"]
}
...

In Step 2, you deployed the training-admin policy in the education/training namespace. The path is relative to the working namespace. So, if you want to create the training-admin policy in the education namespace instead, the paths starts with training/ rather than education/training/.

# Manage namespaces
path "training/sys/namespaces/*" {
   capabilities = ["create", "read", "update", "delete", "list", "sudo"]
}

# Manage policies
path "training/sys/policies/*" {
   capabilities = ["create", "read", "update", "delete", "list", "sudo"]
}
...

Help and Reference