Getting Started: Jobs | Nomad

Jobs are the primary configuration that users interact with when using Nomad. A job is a declarative specification of tasks that Nomad should run. Jobs have a globally unique name, one or many task groups, which are themselves collections of one or many tasks.

The format of the jobs is documented in the job specification. They can either be specified in HashiCorp Configuration Language or JSON, however we recommend only using JSON when the configuration is generated by a machine.

Note: The job created by running nomad job init uses the Docker task driver. To run it, you will need a Nomad client available with Docker installed. This is also true when running Nomad as a dev agent using the -dev flag.

»Running a Job

To get started, use the job init command which generates a skeleton job file:

$ nomad job init
Example job file written to example.nomad

You can view the contents of this file by running cat example.nomad. This example job file declares a single task named redis, which uses the Docker driver to run the a Redis container.

The primary way you interact with Nomad is with the job run command. The run command takes a job file and registers it with Nomad. This is used both to register new jobs and to update existing jobs.

You can register the example job now:

$ nomad job run example.nomad
==> Monitoring evaluation "13ebb66d"
    Evaluation triggered by job "example"
    Allocation "883269bf" created: node "e42d6f19", group "cache"
    Evaluation within deployment: "b0a84e74"
    Evaluation status changed: "pending" -> "complete"
==> Evaluation "13ebb66d" finished with status "complete"

If the output of the nomad job run example.nomad command contains the following message, verify that Docker is installed on your Nomad client nodes or -dev agent node.

Task Group "cache" (failed to place 1 allocation):
  * Constraint "missing drivers": 1 nodes excluded by filter

This message indicates that there are no Nomad clients available running the Docker task driver. This is usually because Docker is stopped or not installed.

Anytime a job is updated, Nomad creates an evaluation to determine what actions need to take place. In this case, because this is a new job, Nomad has determined that an allocation should be created and has scheduled it on your local agent.

To inspect the status of your job you use the status command:

$ nomad status example
ID            = example
Name          = example
Submit Date   = 08/31/19 22:58:40 UTC
Type          = service
Priority      = 50
Datacenters   = dc1
Status        = running
Periodic      = false
Parameterized = false

Summary
Task Group  Queued  Starting  Running  Failed  Complete  Lost
cache       0       0         1        0       0         0

Latest Deployment
ID          = b0a84e74
Status      = successful
Description = Deployment completed successfully

Deployed
Task Group  Desired  Placed  Healthy  Unhealthy
cache       1        1       1        0

Allocations
ID        Node ID   Task Group  Version  Desired  Status   Created  Modified
8ba85cef  171a583b  cache       0        run      running  5m ago   5m ago

Here, the output indicates that the result of your evaluation was the creation of an allocation that is now running on the local node.

An allocation represents an instance of Task Group placed on a node. To inspect an allocation, use the alloc status command:

$ nomad alloc status 8ba85cef
ID                  = 8ba85cef
Eval ID             = 13ebb66d
Name                = example.cache[0]
Node ID             = e42d6f19
Job ID              = example
Job Version         = 0
Client Status       = running
Client Description  = <none>
Desired Status      = run
Desired Description = <none>
Created             = 5m ago
Modified            = 5m ago
Deployment ID       = fa882a5b
Deployment Health   = healthy

Task "redis" is "running"
Task Resources
CPU        Memory           Disk     IOPS  Addresses
8/500 MHz  6.3 MiB/256 MiB  300 MiB  0     db: 127.0.0.1:22672

Task Events:
Started At     = 08/31/19 22:58:49 UTC
Finished At    = N/A
Total Restarts = 0
Last Restart   = N/A

Recent Events:
Time                   Type        Description
08/31/19 22:58:49 UTC  Started     Task started by client
08/31/19 22:58:40 UTC  Driver      Downloading image redis:3.2
08/31/19 22:58:40 UTC  Task Setup  Building Task Directory
08/31/19 22:58:40 UTC  Received    Task received by client

You can see that Nomad reports the state of the allocation as well as its current resource usage. By supplying the -stats flag, more detailed resource usage statistics will be reported.

To see the logs of a task, use the alloc logs command:

$ nomad alloc logs 8ba85cef redis
                 _._
            _.-``__ ''-._
       _.-``    `.  `_.  ''-._           Redis 3.2.1 (00000000/0) 64 bit
   .-`` .-```.  ```\/    _.,_ ''-._
  (    '      ,       .-`  | `,    )     Running in standalone mode
  |`-._`-...-` __...-.``-._|'` _.-'|     Port: 6379
  |    `-._   `._    /     _.-'    |     PID: 1
   `-._    `-._  `-./  _.-'    _.-'
  |`-._`-._    `-.__.-'    _.-'_.-'|
  |    `-._`-._        _.-'_.-'    |           http://redis.io
   `-._    `-._`-.__.-'_.-'    _.-'
  |`-._`-._    `-.__.-'    _.-'_.-'|
  |    `-._`-._        _.-'_.-'    |
   `-._    `-._`-.__.-'_.-'    _.-'
       `-._    `-.__.-'    _.-'
           `-._        _.-'
               `-.__.-'
...

»Modifying a Job

The definition of a job is not static, and is meant to be updated over time. You may update a job to change the docker container, to update the application version, or to change the count of a task group to scale with load.

For now, edit the example.nomad file to update the count and set it to 3. This line is in the cache section around line 145.

# The "count" parameter specifies the number of the task groups that should
# be running under this group. This value must be non-negative and defaults
# to 1.
count = 3

After modifying the job specification, use the job plan command to invoke a dry-run of the scheduler to see what would happen if you ran the updated job:

$ nomad job plan example.nomad
+/- Job: "example"
+/- Task Group: "cache" (2 create, 1 in-place update)
  +/- Count: "1" => "3" (forces create)
      Task: "redis"

Scheduler dry-run:
- All tasks successfully allocated.

Job Modify Index: 7
To submit the job with version verification run:

nomad job run -check-index 7 example.nomad

When running the job with the check-index flag, the job will only be run if the
job modify index given matches the server-side version. If the index has
changed, another user has modified the job and the plan's results are
potentially invalid.

You can see that the scheduler detected the change in count and informs us that it will cause 2 new instances to be created. The in-place update that will occur is to push the updated job specification to the existing allocation and will not cause any service interruption. You can then use the nomad run command provided in the plan output to run the job with version verification.

By running with the -check-index flag, Nomad checks that the job has not been modified since the plan was run. This is useful if multiple people are interacting with the job at the same time to ensure the job hasn’t changed before you apply your modifications.

$ nomad job run -check-index 7 example.nomad
==> Monitoring evaluation "93d16471"
    Evaluation triggered by job "example"
    Evaluation within deployment: "0d06e1b6"
    Allocation "3249e320" created: node "e42d6f19", group "cache"
    Allocation "453b210f" created: node "e42d6f19", group "cache"
    Allocation "883269bf" modified: node "e42d6f19", group "cache"
    Evaluation status changed: "pending" -> "complete"
==> Evaluation "93d16471" finished with status "complete"

Because you set the count of the task group to three, Nomad created two additional allocations to get to the desired state. It is idempotent to run the same job specification again and no new allocations will be created.

Now, change the job to do an application update. In this case, you will change the version of Redis you want to run. Edit the example.nomad file and change the Docker image from “redis:3.2” to “redis:4.0”. This is located around line 261.

# Configure Docker driver with the image
config {
    image = "redis:4.0"
}

When running Nomad without Consul

Because the sample job contains a Consul health check, Nomad’s deployment watcher will wait for the check to pass by default. This will cause your deployment to stall after the first allocation updates. Resolve this by adding the following attribute inside of the update stanza.

health_check = "task_states"

You can run plan again to see what will happen if you submit this change:

$ nomad job plan example.nomad
+/- Job: "example"
+/- Task Group: "cache" (1 create/destroy update, 2 ignore)
  +/- Task: "redis" (forces create/destroy update)
    +/- Config {
      +/- image:           "redis:3.2" => "redis:4.0"
          port_map[0][db]: "6379"
        }

Scheduler dry-run:
- All tasks successfully allocated.

Job Modify Index: 1127
To submit the job with version verification run:

nomad job run -check-index 1127 example.nomad

When running the job with the check-index flag, the job will only be run if the
job modify index given matches the server-side version. If the index has
changed, another user has modified the job and the plan's results are
potentially invalid.

The plan output shows us that one allocation will be updated and that the other two will be ignored. This is due to the max_parallel setting in the update stanza, which is set to 1 to instruct Nomad to perform only a single change at a time.

Once ready, use run to push the updated specification:

$ nomad job run example.nomad
==> Monitoring evaluation "293b313a"
    Evaluation triggered by job "example"
    Evaluation within deployment: "f4047b3a"
    Allocation "27bd4a41" created: node "e42d6f19", group "cache"
    Evaluation status changed: "pending" -> "complete"
==> Evaluation "293b313a" finished with status "complete"

After running, the rolling update can be followed by running nomad status example and watching Allocations section at the bottom. A successful deployment will show an updated version number for Version, a Desired value of run, and a Status of running.

$ nomad status example
ID            = example
Name          = example
Submit Date   = 2019-10-23T18:46:04Z
Type          = service
Priority      = 50
Datacenters   = dc1
Status        = running
Periodic      = false
Parameterized = false

Summary
Task Group  Queued  Starting  Running  Failed  Complete  Lost
cache       0       0         3        0       0         0

Latest Deployment
ID          = b94865ff
Status      = running
Description = Deployment is running

Deployed
Task Group  Desired  Placed  Healthy  Unhealthy  Progress Deadline
cache       3        1       0        0          2019-10-23T18:56:04Z

Allocations
ID        Node ID   Task Group  Version  Desired  Status   Created    Modified
fb621b7f  b750b455  cache       2        run      running  1s ago     1s ago
f3fbf5a8  b750b455  cache       1        run      running  58s ago    47s ago
77549295  b750b455  cache       1        run      running  1m43s ago  48s ago

You can see that Nomad handled the update in three phases, only updating a single allocation in each phase and waiting for it to be healthy for min_healthy_time of 10 seconds before moving on to the next. The update strategy can be configured, but rolling updates makes it less complex to update an application at large scale.

»Stopping a Job

So far you’ve created, run and modified a job. The final step in a job lifecycle is stopping the job. This is done with the job stop command:

$ nomad job stop example
==> Monitoring evaluation "6d4cd6ca"
    Evaluation triggered by job "example"
    Evaluation within deployment: "f4047b3a"
    Evaluation status changed: "pending" -> "complete"
==> Evaluation "6d4cd6ca" finished with status "complete"

When you stop a job, it creates an evaluation which is used to stop all the existing allocations. If you now query the job status, you can see it is now marked as dead (stopped), indicating that the job has been stopped and Nomad is no longer running it:

$ nomad status example
ID            = example
Name          = example
Submit Date   = 08/31/19 17:30:40 UTC
Type          = service
Priority      = 50
Datacenters   = dc1
Status        = dead (stopped)
Periodic      = false
Parameterized = false

Summary
Task Group  Queued  Starting  Running  Failed  Complete  Lost
cache       0       0         0        0       6         0

Latest Deployment
ID          = f4047b3a
Status      = successful
Description = Deployment completed successfully

Deployed
Task Group  Desired  Placed  Healthy  Unhealthy
cache       3        3       3        0

Allocations
ID        Node ID   Task Group  Version  Desired  Status    Created    Modified
8ace140d  2cfe061e  cache       2        stop     complete  5m ago     5m ago
8af5330a  2cfe061e  cache       2        stop     complete  6m ago     6m ago
df50c3ae  2cfe061e  cache       2        stop     complete  6m ago     6m ago

If you wanted to start the job again, you can run it again.

Users of Nomad primarily interact with jobs, and you’ve now seen how to create and scale our job, perform an application update, and do a job tear down. Next you will add another Nomad client to create our first cluster.

Stop the Nomad agent with Ctrl-C before moving on to the next section.

Infrastructure

Security

Networking

Applications

»Running a Job

»Modifying a Job

»Stopping a Job