The Terraform Cloud Business Tier allows you to manage isolated, private, or on-premises infrastructure using self-hosted Terraform Cloud agents. The agent polls Terraform Cloud or Terraform Enterprise for any changes to your configuration and executes the changes locally, so you do not need to allow public ingress traffic to your resources. Agents allow you to control infrastructure in private environments without modifying your network perimeter.
In this tutorial, you will create a Terraform Cloud agent pool, launch local Dockerized agents on your machine, and configure a Terraform Cloud workspace to use the agents to manage an Nginx container on your machine. You will also manage tokens for multiple Terraform Cloud agents. By the end of the tutorial, you will understand how to set up a similar configuration in your own environment.
Note: Agents are only available for the Business tier of Terraform Cloud. If you are interested in upgrading, contact a sales representative.
»Prerequisites
For this tutorial, you will need:
- A Terraform Cloud account with the Business upgrade
- A GitHub account
- Docker
»Create an agent pool
Agent pools are groups of agents that can share tokens. When you associate a workspace with an agent pool, any agent in the pool can execute a run in that workspace.
To create an agent pool, navigate to the "Agents" panel within your Terraform Cloud's "Settings" page and click "Create agent pool."
Enter education as the name, then click "Continue."
Next, Terraform Cloud will prompt you to generate a token for the agent pool.
Add agent1 as the description and click "Create token."
The next page displays your agent token and example commands for getting your agent running. This page only displays the token on creation — leave it open for now.
The page suggests a command to run the Docker agent that passes in environment variables
with individual -e flags. In this tutorial, you will use the --env-file
parameter instead, which makes it easier to manage multiple variables within a
single file, so you will not have to re-enter them each time you need to relaunch
the container. Using an environment variable file also prevents credentials
from appearing in your process tables, granting an extra layer of security.
You can configure multiple tokens per agent pool, or have one shared token across all agents in the pool. By using unique tokens, you can revoke the token of one agent and stop its work without disrupting other agents in the pool, which you will do later in this tutorial.
»Configure a workspace to use agents
Now configure a version-control-driven Terraform Cloud workspace to use the agent pool.
In a new browser window, create a fork of the demo repository in your own GitHub account. This repository contains sample configuration to spin up an Nginx Docker container.
In that same browser window, create a new
learn-terraform-cloud-agents workspace associated with your forked
repository. For a detailed example of how to create a workspace, revisit the
Create a Workspace
tutorial.
Next, navigate to your workspace's "General" settings.
Under "Execution Mode," select "Agent" and select "education" in the drop down under "Agent Pool."
Make sure to click "Save Settings" at the bottom of the page.
»Launch the agent
Create a file called agent1.list file and paste in the contents below. Update
the value of TFC_AGENT_TOKEN with the token you created in the previous step.
Name the agent agent1 for this
exercise. This name will show up in the agent management UI and on runs,
allowing you to identify specific agents in the future. If you plan on using
unique tokens for each agent, use the same name for the agent and token for
clarity.
The only required environment variable is TFC_AGENT_TOKEN, but the agent
loads all variables in your Docker environment. You can use this as an
alternative to storing credentials and environment variables in your Terraform
Cloud Workspace's configuration.
Start your first containerized agent on your local machine. For this exercise,
you are mounting the Docker socket using -v /var/run/docker.sock:/var/run/docker.sock. Mounting the socket allows the
containerized agent to use the Docker provider to manage other containers on
your machine.
Keep this terminal open.
Because this scenario uses the containerized Terraform to manage other Docker
containers, the agent container needs to access the Docker socket. The
Terraform Cloud Agent runs as the non-root tfc-agent user within the
container, so you need to explicitly modify the permissions for the Docker socket.
Open a new terminal window.
First, verify the group ID of the docker group on your system. It will be the third field in the output returned, in this case 281.
Exec into the agent container as the root user.
Create a docker group within the container, replacing <GROUP_ID> with the docker group ID of your host.
Add the tfc-agent user, which owns the agent process, to the docker group.
Exit out of the container.
Note: Mounting the Docker socket and manipulating its permissions is only necessary if you are running the Dockerized agent and need it to manage other Docker containers on the host machine.
Once the agent container launches, verify that it has registered with the pool in the Terraform Cloud interface.
Notice "1 out of 5 purchased agents" next to "Agent Pools" — the number of agents available to you is determined by your Terraform Cloud for Business plan.
Agents are available as Docker containers and as standalone x86 binaries. For guidance on using the binaries, refer to the agent documentation.
»Create a local Nginx container
In your browser, navigate back to your Terraform Cloud learn-terraform-cloud-agents workspace and
trigger a run by clicking "Queue plan."
The run "Execution Mode" shows that it is running in your local Docker agent. The plan details show the names of the agent pool and agent responsible for the run.
In your terminal, the Docker agent logs display the agent's Terraform actions.
Approve the proposed changes in Terraform Cloud by clicking on "Confirm & Apply," and then confirming the plan.
The agent will now execute the work. The agent is self-contained and will not spin up additional containers for the Terraform execution itself — it downloads the Terraform version your configuration specifies and any providers necessary to complete the work task. In this tutorial, the agent launches an additional Docker container because that is the resource declared in the sample Terraform configuration.
Once the apply is complete, open a new terminal window and confirm that your Nginx container is running by
using docker ps or visiting localhost:8000.
Though this tutorial targets your local machine, you can use the same configuration on any instance of your infrastructure. While running, the agent polls Terraform Cloud for any new workloads it needs to complete. Since this is a pull-based pattern, you only need to allow TCP/443 egress traffic and DNS resolution. This is a lightweight way to use Terraform Cloud within your isolated network segments without needing to configure your own installation of Terraform Enterprise.
»Configure a second agent
Each agent is single-threaded and can only execute the work of one run at a time. For concurrent workloads, you must provision multiple agents. Agents within a pool can share tokens, or use unique ones so you can easily revoke the token of one agent without disrupting others.
Launch a second agent with a unique token.
- Navigate back to your Terraform Cloud organization settings.
- Select the "Agents" panel.
- Click on the "education" agent pool.
- Click "+ New token."
- Name the new agent
agent2.
This will take you through the same steps as your first token creation. You will use this second token to launch another agent.
In a new terminal, create a file named agent2.list and open it. Copy and
paste the contents below, and set the value of TFC_AGENT_TOKEN to be your new
token.
Now launch your second agent.
The "Agents" page in Terraform Cloud will now display 2 agents ready to retrieve and process workloads.
Any other containers that have exited will appear here as well, but do not count toward your allotted number of agents. Refer to the Terraform Cloud Agent documentation for a detailed list of possible agent statuses and how they count toward your agent totals.
You can configure additional agent pools as well, allowing you to maintain agents in each of your data centers and network segments.
»Revoke a token
Next, revoke the token of one of your agents. On the education agent
pool page, click "Revoke Token" for the agent2 token.
Once you revoke the token, the agent2 container will log its graceful
shutdown and exit.
Meanwhile, the agent1 container will continue running. This demonstrates the
advantage of using unique tokens for your agents: you can revoke a token
without disrupting any other agents. The state of agent2 will change in the "Agents"
dashboard in Terraform Cloud.
The exited agent is in an "Unknown" state but will expire out of the list and your agent allocation count within 2 hours. If this agent comes back online before that timeout, it will return to an "Idle" state. Only agents in the "Idle," "Busy," or "Unknown" states count against your purchased agent limit.
»Destroy resources
Now that you have created and used Terraform Cloud agents to create resources in an isolated environment, clean up all resources created in this tutorial.
»Destroy the infrastructure
Prior to removing the agent pool, remove the resources created by queueing a destroy plan in your workspace. Follow the Destroy Infrastructure step of the clean up tutorial for step-by-step instructions if needed.
Your agent container will log the destruction plan as well.
Note: Destroy your infrastructure resources prior to destroying the agent pool. Terraform Cloud will not be able to destroy your local containers without the agent.
»Disassociate the agent pool
To delete an agent pool, you must first disassociate it from all workspaces. In your workspace settings, change the "Execution Mode" to "Remote." Be sure to click "Save Settings."
»Delete the agent pool
On the "Agent Pool" page in your Terraform Cloud organization settings, scroll to the bottom for the "Delete Agent Pool" section. Click "Delete agent pool" on the "Agents" page and confirm the prompt "Yes, delete agent pool."
»Next steps
Terraform Cloud's self-hosted agents allow you to manage more of your resources using Terraform Cloud or Terraform Enterprise without modifying your network perimeter. Since run execution modes are workspace-specific, you can continue using remote execution for any publicly accessible-resources and use the self-hosted agent for resources that require extra security.
Learn more in the Terraform Cloud Agents documentation.
Review the configuration details and differences for using Terraform Cloud Agents with Terraform Enterprise.
Read about other Business Tier features.
Explore the Terraform Cloud/Enterprise provider.
