Governance and Policy

Governance & Policy on Nomad

Nomad Enterprise is aimed at teams and organizations and addresses the organizational complexity of multi-team and multi-cluster deployments with collaboration and governance features.

This track provides best practices and guidance for operating Nomad securely in a multi-team setting through features such as namespaces, resource quotas, and Sentinel.

Namespaces

Namespaces allow a single cluster to be shared by many teams and projects without conflict. Nomad requires unique job IDs within namespaces but not across namespaces, this allows each team to operate independently of others.

When combined with ACLs, the isolation of namespaces can be enforced, only allowing designated users access to read or modify the jobs and associated objects in a namespace.

When resource quotas are applied to a namespace they provide a means to limit resource consumption by the jobs in the namespace. This can prevent a single actor from consuming excessive cluster resources and negatively impacting other teams and applications sharing the cluster.

Resource Quotas

When many teams or users are sharing Nomad clusters, there is the concern that a single user could use more than their fair share of resources. Resource quotas provide a mechanism for cluster administrators to restrict the resources within a namespace.

Quota specifications are first class objects in Nomad. A quota specification has a unique name, an optional human readable description, and a set of quota limits. The quota limits define the allowed resource usage within a region.

Quota objects are shareable among namespaces. This allows an operator to define higher level quota specifications, such as a prod-api quota, and multiple namespaces can apply the prod-api quota specification.

Sentinel

Sentinel Overview

  • Sentinel Policies - Policies are able to introspect on request arguments and use complex logic to determine if the request meets policy requirements. For example, a Sentinel policy may restrict Nomad jobs to only using the "docker" driver or prevent jobs from being modified outside of business hours.

  • Policy Scope - Sentinel policies declare a "scope", which determines when the policies apply. Currently the only supported scope is "submit-job", which applies to any new jobs being submitted, or existing jobs being updated.

  • Enforcement Level - Sentinel policies support multiple enforcement levels. The advisory level emits a warning when the policy fails, while soft-mandatory and hard-mandatory will prevent the operation. A soft-mandatory policy can be overridden if the user has necessary permissions.

Sentinel policies

Each Sentinel policy has a unique name, an optional description, applicable scope, enforcement level, and a Sentinel rule definition. If multiple policies are installed for the same scope, all of them are enforced and must pass.

Sentinel policies cannot be used unless the ACL system is enabled.

Policy scope

Sentinel policies specify an applicable scope, which limits when the policy is enforced. This allows policies to govern various aspects of the system.

The following table summarizes the scopes that are available for Sentinel policies:

ScopeDescription
submit-jobApplies to any jobs (new or updated) being registered

Enforcement Level

Sentinel policies specify an enforcement level which changes how a policy is enforced. This allows for more flexibility in policy enforcement.

The following table summarizes the enforcement levels that are available:

Enforcement LevelDescription
advisoryIssues a warning when a policy fails
soft-mandatoryPrevents operation when a policy fails, issues a warning if overridden
hard-mandatoryPrevents operation when a policy fails

The sentinel-override capability is required to override a soft-mandatory policy. This allows a restricted set of users to have override capability when necessary.

Multi-Region configuration

Nomad supports multi-datacenter and multi-region configurations. A single region is able to service multiple datacenters, and all servers in a region replicate their state between each other. In a multi-region configuration, there is a set of servers per region. Each region operates independently and is loosely coupled to allow jobs to be scheduled in any region and requests to flow transparently to the correct region.

When ACLs are enabled, Nomad depends on an "authoritative region" to act as a single source of truth for ACL policies, global ACL tokens, and Sentinel policies. The authoritative region is configured in the server stanza of agents, and all regions must share a single authoritative source. Any Sentinel policies are created in the authoritative region first. All other regions replicate Sentinel policies, ACL policies, and global ACL tokens to act as local mirrors. This allows policies to be administered centrally, and for enforcement to be local to each region for low latency.

Next Steps

Click the "Next" button below to continue to Namespaces, or you can jump ahead to other topics by clicking the links below: