Start Nomad and Run Your First Job | Nomad

Nomad relies on a long running agent on every machine in the cluster. The agent can run either in server or client mode. The cluster servers are responsible for managing the cluster. All other agents in the cluster should be in client mode. A Nomad client is a very lightweight process that registers the host machine, performs heart-beating, and runs the tasks that are assigned to it by the servers. The agent must be run on every node that is part of the cluster so that the servers can assign work to those machines.

In this guide, you start the Nomad agent in development mode also called a dev agent. Dev agents quickly start a single agent that acts as a client and server to test job configurations or prototype interactions. You also use the nomad command-line tool to discover the status and server membership for your agent. Finally, you create and start a sample job using the nomad job init and nomad job run commands.

Warning: A single server deployment is highly discouraged as data loss is inevitable in a failure scenario. Each region must have at least one server, though a cluster of 3 or 5 servers is recommended for production.

»Start the agent

At the end of the Get Started: Install tutorial, you used the nomad command to verify that you could perform the remaining steps in this guide. If you are using Vagrant, make sure that you are in an SSH session to the virtual machine by checking that the prompt is vagrant@nomad:$.

Now, use the sudo command to start a single Nomad agent in development mode with the nomad agent command. Typically any agent in client mode must be started with root level privilege. Nomad makes use of operating system primitives for resource isolation which require elevated permissions. The agent functions as non-root, but certain task drivers are not available.

Add the -bind flag and set the value to 0.0.0.0. Also, use the -log-level flag and set the output log level to INFO. By default, Nomad dev agents bind to 127.0.0.1 which causes issues with Vagrant port forwarding. Nomad dev agents also default to DEBUG level logging, which can be a bit noisy.

Note: The nomad agent -dev command should only be used for experimental environments because it does not persist state between runs.

$ sudo nomad agent -dev -bind 0.0.0.0 -log-level INFO

==> No configuration files loaded
==> Starting Nomad agent...
==> Nomad agent configuration:

       Advertise Addrs: HTTP: 10.0.2.15:4646; RPC: 10.0.2.15:4647; Serf: 10.0.2.15:4648
            Bind Addrs: HTTP: 0.0.0.0:4646; RPC: 0.0.0.0:4647; Serf: 0.0.0.0:4648
                Client: true
             Log Level: INFO
                Region: global (DC: dc1)
                Server: true
               Version: 1.0.1

==> Nomad agent started! Log data will stream in below:

    2021-02-08T17:43:19.126Z [INFO]  agent: detected plugin: name=exec type=driver plugin_version=0.1.0
    2021-02-08T17:43:19.126Z [INFO]  agent: detected plugin: name=qemu type=driver plugin_version=0.1.0
    2021-02-08T17:43:19.126Z [INFO]  agent: detected plugin: name=java type=driver plugin_version=0.1.0
    2021-02-08T17:43:19.126Z [INFO]  agent: detected plugin: name=docker type=driver plugin_version=0.1.0
    2021-02-08T17:43:19.126Z [INFO]  agent: detected plugin: name=raw_exec type=driver plugin_version=0.1.0
    2021-02-08T17:43:19.126Z [INFO]  agent: detected plugin: name=nvidia-gpu type=device plugin_version=0.1.0
    2021-02-08T17:43:19.127Z [INFO]  nomad.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:10.0.2.15:4647 Address:10.0.2.15:4647}]"
    2021-02-08T17:43:19.128Z [INFO]  nomad: serf: EventMemberJoin: nomad.global 10.0.2.15
    2021-02-08T17:43:19.128Z [INFO]  nomad: starting scheduling worker(s): num_workers=1 schedulers=[batch, system, service, _core]
    2021-02-08T17:43:19.128Z [INFO]  client: using state directory: state_dir=/tmp/NomadClient609060854
    2021-02-08T17:43:19.128Z [INFO]  client: using alloc directory: alloc_dir=/tmp/NomadClient440492765
    2021-02-08T17:43:19.129Z [INFO]  client.fingerprint_mgr.cgroup: cgroups are available
    2021-02-08T17:43:19.130Z [INFO]  nomad.raft: entering follower state: follower="Node at 10.0.2.15:4647 [Follower]" leader=
    2021-02-08T17:43:19.130Z [INFO]  nomad: adding server: server="nomad.global (Addr: 10.0.2.15:4647) (DC: dc1)"
    2021-02-08T17:43:19.135Z [INFO]  client.fingerprint_mgr.consul: consul agent is available
    2021-02-08T17:43:19.140Z [WARN]  client.fingerprint_mgr.network: unable to parse speed: path=/sbin/ethtool device=lo
    2021-02-08T17:43:19.142Z [WARN]  client.fingerprint_mgr.network: unable to parse speed: path=/sbin/ethtool device=lo
    2021-02-08T17:43:19.143Z [WARN]  client.fingerprint_mgr.network: unable to parse speed: path=/sbin/ethtool device=docker0
    2021-02-08T17:43:21.128Z [WARN]  nomad.raft: heartbeat timeout reached, starting election: last-leader=
    2021-02-08T17:43:21.128Z [INFO]  nomad.raft: entering candidate state: node="Node at 10.0.2.15:4647 [Candidate]" term=2
    2021-02-08T17:43:21.129Z [INFO]  nomad.raft: election won: tally=1
    2021-02-08T17:43:21.129Z [INFO]  nomad.raft: entering leader state: leader="Node at 10.0.2.15:4647 [Leader]"
    2021-02-08T17:43:21.129Z [INFO]  nomad: cluster leadership acquired
    2021-02-08T17:43:21.130Z [INFO]  nomad.core: established cluster id: cluster_id=610b66a1-1f3c-6803-bc55-65bdb5a2a329 create_time=1612806201130151018
    2021-02-08T17:43:23.148Z [INFO]  client.plugin: starting plugin manager: plugin-type=csi
    2021-02-08T17:43:23.148Z [INFO]  client.plugin: starting plugin manager: plugin-type=driver
    2021-02-08T17:43:23.148Z [INFO]  client.plugin: starting plugin manager: plugin-type=device
    2021-02-08T17:43:23.169Z [INFO]  client: started client: node_id=3825e530-a0c4-9a00-f807-1d11e63f1a48
    2021-02-08T17:43:23.171Z [INFO]  client: node registration complete
    2021-02-08T17:43:24.172Z [INFO]  client: node registration complete
==> Newer Nomad version available: 1.0.3 (currently running: 1.0.1)

The output indicates the agent has started in both server and client mode. Wait to continue to the next section until you see the agent has acquired leadership.

During startup, you might even see a log line that tells you that there is a new version of Nomad available.

==> Newer Nomad version available: 1.0.3 (currently running: 1.0.1)

This log line is informative and doesn't have any impact on the currently running agent.

»Discover agent information

Start another terminal session. When using Vagrant, start a new local shell and then re-run the vagrant ssh command from your nomad-demo folder to get back to the vagrant@nomad:$ prompt.

$ vagrant ssh

In that session, use nomad node status to view the registered nodes of the Nomad cluster.

$ nomad node status

Since you only have one local agent, your output should only include one node.

ID        DC   Name   Class   Drain  Eligibility  Status
f9381a0b  dc1  nomad  <none>  false  eligible     ready

The output shows your Node ID, its datacenter, node name, node class, drain mode and current status. The Node ID is a randomly generated UUID.

Notice that your node is in the ready state.

The agent is also in server mode, which means it is part of the gossip protocol used to connect all the server instances together. You can view the members of the gossip ring using the server members command.

$ nomad server members

Since you only have one local agent, your output should only include one node.

Name          Address    Port  Status  Leader  Protocol  Build  Datacenter  Region
nomad.global  127.0.0.1  4648  alive   true    2         1.0.1  dc1         global

The output shows your agent, the address it is running on, its health state, some version information, and the datacenter and region. Additional metadata can be viewed by providing the -detailed flag.

»Run your first job

Jobs are the primary configuration that users interact with when using Nomad. A job is a declarative specification of tasks that Nomad should run. Jobs have a globally unique name and one or many task groups, which are themselves collections of one or many tasks.

The format of the jobs is documented in the job specification. They can either be specified in HashiCorp Configuration Language or JSON; however, you should only use JSON when the configuration is generated by a machine.

Note: The job created by running nomad job init uses the Docker task driver. To run it, you need a Nomad client available with Docker installed. This is also true when running Nomad as a dev-agent using the -dev flag. Also, the user that you are running Nomad as needs to be a member of the docker group.

»Generate the sample job

To get started, use the nomad job init command which generates a well-documented sample job file:

$ nomad job init

Example job file written to example.nomad

You can view the contents of this file by running cat example.nomad. This example job file declares a single task named redis, which uses the Docker driver to run the a Redis container.

Did you know? You can use the -short flag of the nomad job init command to create a similar job specification without comments or unnecessary stanzas. You can also override the filename created by supplying it as the last argument to the command.

The primary way you interact with Nomad is with the nomad job run command. The run command takes a job file and registers it with Nomad. This is used both to register new jobs and to update existing jobs.

$ nomad job run example.nomad

The command outputs information about the scheduling process.

==> Monitoring evaluation "6792e7a6"
    Evaluation triggered by job "example"
    Evaluation within deployment: "99297d1d"
    Allocation "635a981a" created: node "3825e530", group "cache"
    Evaluation status changed: "pending" -> "complete"
==> Evaluation "6792e7a6" finished with status "complete"

Here, the output indicates that the result evaluating your job submission was the creation of an allocation that is now running on the local node.

If the output of the nomad job run example.nomad command contains the following message, verify that Docker is installed on your Nomad client nodes or -dev agent node. The Vagrant environment provides this automatically for you.

Task Group "cache" (failed to place 1 allocation):
  * Constraint "missing drivers": 1 nodes excluded by filter

This message indicates that there are no Nomad clients available running the Docker task driver. This is typically because Docker is stopped or not installed.

Anytime a job is updated, Nomad creates an evaluation to determine what actions need to take place. Because this is a new job, Nomad determined that an allocation should be created and has scheduled it on your local agent.

Use the nomad job status command to inspect the status of your job:

$ nomad job status example

ID            = example
Name          = example
Submit Date   = 2021-02-08T18:13:14Z
Type          = service
Priority      = 50
Datacenters   = dc1
Namespace     = default
Status        = running
Periodic      = false
Parameterized = false

Summary
Task Group  Queued  Starting  Running  Failed  Complete  Lost
cache       0       0         1        0       0         0

Latest Deployment
ID          = 99297d1d
Status      = successful
Description = Deployment completed successfully

Deployed
Task Group  Desired  Placed  Healthy  Unhealthy  Progress Deadline
cache       1        1       1        0          2021-02-08T18:23:31Z

Allocations
ID        Node ID   Task Group  Version  Desired  Status   Created  Modified
635a981a  3825e530  cache       0        run      running  21s ago  3s ago

When Nomad runs a job it creates allocations based on the task groups within the job. To inspect an allocation, use the nomad alloc status command. Replace the allocation ID in the command with the allocation ID you received when running nomad job status example in the preceding step.

$ nomad alloc status 635a981a

ID                  = 635a981a-8a17-b344-15c4-8959a5c7e7fa
Eval ID             = 6792e7a6
Name                = example.cache[0]
Node ID             = 3825e530
Node Name           = nomad
Job ID              = example
Job Version         = 0
Client Status       = running
Client Description  = Tasks are running
Desired Status      = run
Desired Description = 
Created             = 58s ago
Modified            = 40s ago
Deployment ID       = 99297d1d
Deployment Health   = healthy

Allocation Addresses
Label  Dynamic  Address
*db    yes      127.0.0.1:24859 -> 6379

Task "redis" is "running"
Task Resources
CPU        Memory           Disk     Addresses
2/500 MHz  988 KiB/256 MiB  300 MiB

Task Events:
Started At     = 2021-02-08T18:13:21Z
Finished At    = N/A
Total Restarts = 0
Last Restart   = N/A

Recent Events:
Time                  Type        Description
2021-02-08T18:13:21Z  Started     Task started by client
2021-02-08T18:13:14Z  Driver      Downloading image
2021-02-08T18:13:14Z  Task Setup  Building Task Directory
2021-02-08T18:13:14Z  Received    Task received by client

Nomad reports the state of the allocation as well as its current resource usage.

»Fetch the job logs

Nomad automatically captures the information written to stdout and stderr into log files for each task defined in the job specification. You can fetch these logs using the nomad alloc logs command. To see the stderr log, you must supply the -stderr flag before the allocation ID and task name in the command.

Run the nomad alloc logs command to fetch the logs from the redis task in your allocation, again substituting your allocation ID into the proper place in the command.

$ nomad alloc logs 635a981a redis

The command returns the information written to stdout by the redis task.

1:C 08 Feb 16:40:59.822 # Warning: no config file specified, using the default config. In order to specify a config file use redis-server /path/to/redis.conf
                _._
           _.-``__ ''-._
      _.-``    `.  `_.  ''-._           Redis 3.2.12 (00000000/0) 64 bit
  .-`` .-```.  ```\/    _.,_ ''-._
 (    '      ,       .-`  | `,    )     Running in standalone mode
 |`-._`-...-` __...-.``-._|'` _.-'|     Port: 6379
 |    `-._   `._    /     _.-'    |     PID: 1
  `-._    `-._  `-./  _.-'    _.-'
 |`-._`-._    `-.__.-'    _.-'_.-'|
 |    `-._`-._        _.-'_.-'    |           http://redis.io
  `-._    `-._`-.__.-'_.-'    _.-'
 |`-._`-._    `-.__.-'    _.-'_.-'|
 |    `-._`-._        _.-'_.-'    |
  `-._    `-._`-.__.-'_.-'    _.-'
      `-._    `-.__.-'    _.-'
          `-._        _.-'
              `-.__.-'

...

Congratulations! You're running a redis container with Nomad.

»Next steps

In this tutorial you started a local Nomad agent in development node. You then used the nomad command line tool to discover information about your agent. Finally, you generated a sample job using the nomad job init command and submitted it to your Nomad agent.

Leave this example job running, because you learn how to modify, scale, and stop it in the next tutorials.

Infrastructure

Security

Networking

Applications