Deploy Infrastructure with Terraform and CircleCI

CircleCI is a continuous integration and delivery (CI/CD) platform for automating software builds, tests, and deployments. The CI/CD paradigm establishes version control repositories as the source of truth for your deployments. It also helps teams quickly ship new features and fixes by defining pipelines that help ensure the stability and resilience of your services through testing and automation. You can build deployment pipelines of varying complexity to satisfy your organization’s requirements for production deployments.

Using Terraform to manage your infrastructure as code enables the benefits of the CI/CD workflow for infrastructure deployments. Since your infrastructure is codified, your team can collaborate and review it and deploy it using automated pipelines instead of manual orchestration. To automate Terraform operations in a remote environment, you need to configure remote state storage so Terraform can access and manage your project's state across runs.

In this tutorial, you will use CircleCI and Terraform to deploy an S3-backed web application. You will configure and review an automated Terraform workflow and use Terraform Cloud for remote state storage.

»Prerequisites

This tutorial assumes that you are familiar with the Terraform and Terraform Cloud workflows. If you are new to Terraform, complete Get Started collection first. If you are new to Terraform Cloud, complete the Terraform Cloud Get Started tutorials first.

In order to complete this tutorial, you will need the following:

A GitHub account
A CircleCI account. Sign up with your GitHub account so CircleCI can build and deploy from your GitHub repositories. Review the CircleCI getting started guide for an introduction to the workflow if you are unfamiliar.
An AWS account.
A Terraform Cloud account.

»Create a Terraform Cloud token

In order to authenticate with Terraform Cloud to store your project's Terraform state, you need to configure your Terraform Cloud integration with a Terraform Cloud API token.

Navigate to your organization settings, then select the Teams page.

Tip: If you are using a free Terraform Cloud account, create a token for your default Owners team. Otherwise, choose or create a new team with permissions to manage workspaces.

Under the Team API Token section, click Create a team token.

Store this token in a secure place as Terraform Cloud will not display it again. Later in this tutorial, you will set an environment variable in your CircleCI project to this token value for your build to use.

»Fork and clone example configuration

Fork this tutorial’s example repository in GitHub. This repository contains example configuration to deploy Terramino, a Terraform-skinned Tetris game, to an AWS S3 bucket using Terraform and CircleCI.

Once you have forked the repository, clone it to your machine. Edit the command below to include your own GitHub username.

$ git clone https://github.com/YOUR_USER_NAME/learn-terraform-circleci

$ git clone https://github.com/YOUR_USER_NAME/learn-terraform-circleci

Then, navigate to the repository.

$ cd learn-terraform-circleci

$ cd learn-terraform-circleci

»Review Terraform configuration

Open the main.tf file.

The Terraform configuration in this repository deploys Terramino, a Terraform-skinned Tetris application, to an AWS S3 bucket.

 provider "aws" {
   region = var.region

   default_tags {
     tags = {
       hashicorp-learn = "circleci"
     }
   }
 }

 resource "random_uuid" "randomid" {}

 resource "aws_s3_bucket" "app" {
   tags = {
     Name = "App Bucket"
   }

   bucket        = "${var.app}.${var.label}.${random_uuid.randomid.result}"
   force_destroy = true
 }

 resource "aws_s3_bucket_object" "app" {
   acl          = "public-read"
   key          = "index.html"
   bucket       = aws_s3_bucket.app.id
   content      = file("./assets/index.html")
   content_type = "text/html"
 }

 resource "aws_s3_bucket_acl" "bucket" {
   bucket = aws_s3_bucket.app.id
   acl    = "public-read"
 }

 resource "aws_s3_bucket_website_configuration" "terramino" {
   bucket = aws_s3_bucket.app.bucket

   index_document {
     suffix = "index.html"
   }

   error_document {
     key = "error.html"
   }
 }

main.tf

 provider "aws" {
   region = var.region

   default_tags {
     tags = {
       hashicorp-learn = "circleci"
     }
   }
 }

 resource "random_uuid" "randomid" {}

 resource "aws_s3_bucket" "app" {
   tags = {
     Name = "App Bucket"
   }

   bucket        = "${var.app}.${var.label}.${random_uuid.randomid.result}"
   force_destroy = true
 }

 resource "aws_s3_bucket_object" "app" {
   acl          = "public-read"
   key          = "index.html"
   bucket       = aws_s3_bucket.app.id
   content      = file("./assets/index.html")
   content_type = "text/html"
 }

 resource "aws_s3_bucket_acl" "bucket" {
   bucket = aws_s3_bucket.app.id
   acl    = "public-read"
 }

 resource "aws_s3_bucket_website_configuration" "terramino" {
   bucket = aws_s3_bucket.app.bucket

   index_document {
     suffix = "index.html"
   }

   error_document {
     key = "error.html"
   }
 }

This configuration uses a terraform.tfvars file to set values for the input variables of your configuration. Open the terraform.tfvars file to review its contents.

region = "us-east-1"
label  = "hashicorp"
app    = "terramino"

terraform.tfvars

region = "us-east-1"
label  = "hashicorp"
app    = "terramino"

Next, open the terraform.tf file. The configuration in this file defines the required provider and Terraform versions for this configuration. It also includes an empty cloud block.

terraform {
  cloud {}

  required_providers {
    aws = {
      source  = "hashicorp/aws"
      version = "~> 4.4.0"
    }
  }

  required_version = ">= 1.2.0"
}

main.tf

terraform {
  cloud {}

  required_providers {
    aws = {
      source  = "hashicorp/aws"
      version = "~> 4.4.0"
    }
  }

  required_version = ">= 1.2.0"
}

The cloud block configures a Terraform Cloud integration for CLI-driven Terraform Cloud runs. As of Terraform 1.2.0, you can configure the cloud block using environment variables that let you dynamically determine which Terraform Cloud organization or workspace to deploy to. Terraform also lets you use an environment variable to pass a Terraform Cloud token to your CI/CD system. Later in this tutorial, you will configure environment variables in your CircleCI project to configure your Terraform Cloud integration.

»Review CircleCI jobs and workflow

Navigate to the .circleci subdirectory

$ cd .circleci

$ cd .circleci

CircleCI pipelines consist of jobs and workflows. A job is a collection of steps executed together. A workflow is a sequence of jobs that progresses based on the result of the individual steps. For example, your CircleCI configuration may define a testing job that runs tests on your infrastructure and a deployment job that ships your changes. You can then compose these jobs into a workflow, specifying that if the test job succeeds, CircleCI should trigger the deployment job.

Each job declares an executor, which defines the environment CircleCI will perform the job's steps in. All of the jobs in this configuration use the Docker executor and reference a Docker image that contains the latest version of the Terraform binary.

Open the config.yml file in your file editor to review the jobs and workflows for this configuration.

This configuration defines four jobs: plan-apply, apply, plan-destroy, and destroy. It composes these jobs into an automated Terraform workflow.

»Review the `plan-apply` job

The plan-apply job copies your repository using the checkout step, runs terraform init to initialize your configuration, and generates a plan file named tfapply by running terraform plan -out.

The job uses the hashicorp/terraform:light Docker image, which contains the latest version of the Terraform binary. The persist_to_workspace step saves the initialized configuration and your environment variables for use in the following jobs. persist_to_workspace allows you to run this initialized Terraform configuration throughout the rest of the jobs in the workflow as if they are running on the same machine.

jobs:
  plan-apply:
    working_directory: /tmp/project
    docker:
      - image: docker.mirror.hashicorp.services/hashicorp/terraform:light
    steps:
      - checkout
      - run:
          name: terraform init & plan
          command: |
            terraform init -input=false
            terraform plan -out tfapply -var-file terraform.tfvars
      - persist_to_workspace:
          root: .
          paths:
            - .

.circleci/config.yml

jobs:
  plan-apply:
    working_directory: /tmp/project
    docker:
      - image: docker.mirror.hashicorp.services/hashicorp/terraform:light
    steps:
      - checkout
      - run:
          name: terraform init & plan
          command: |
            terraform init -input=false
            terraform plan -out tfapply -var-file terraform.tfvars
      - persist_to_workspace:
          root: .
          paths:
            - .

»Review the `apply` job

The apply job invokes the attach_workspace step to load the persisted workspace from the plan job, giving this job access to its artifacts, including the execution plan captured in the tfapply file. It then runs terraform apply using the -auto-approve flag to avoid prompting for user input.

apply:
  docker:
    - image: docker.mirror.hashicorp.services/hashicorp/terraform:light
  steps:
    - attach_workspace:
        at: .
    - run:
        name: terraform
        command: |
          terraform apply -auto-approve tfapply
    - persist_to_workspace:
        root: .
        paths:
          - .

apply:
  docker:
    - image: docker.mirror.hashicorp.services/hashicorp/terraform:light
  steps:
    - attach_workspace:
        at: .
    - run:
        name: terraform
        command: |
          terraform apply -auto-approve tfapply
    - persist_to_workspace:
        root: .
        paths:
          - .

»Review the `plan-destroy` and `destroy` jobs

Similarly to the plan-apply and apply jobs, the plan-destroy job creates an execution plan and the destroy job executes that saved plan to remove all of the infrastructure tracked in the project's state file. It uses the persisted workspace to access the execution plan across jobs.

plan-destroy:
  docker:
    - image: docker.mirror.hashicorp.services/hashicorp/terraform:light
  steps:
    - attach_workspace:
        at: .
    - run:
        name: terraform create destroy plan
        command: |
          terraform plan -destroy -out tfdestroy -var-file terraform.tfvars
    - persist_to_workspace:
        root: .
        paths:
          - .

destroy:
  docker:
    - image: docker.mirror.hashicorp.services/hashicorp/terraform:light
  steps:
    - attach_workspace:
        at: .
    - run:
        name: terraform destroy
        command: |
          terraform apply -auto-approve tfdestroy

plan-destroy:
  docker:
    - image: docker.mirror.hashicorp.services/hashicorp/terraform:light
  steps:
    - attach_workspace:
        at: .
    - run:
        name: terraform create destroy plan
        command: |
          terraform plan -destroy -out tfdestroy -var-file terraform.tfvars
    - persist_to_workspace:
        root: .
        paths:
          - .

destroy:
  docker:
    - image: docker.mirror.hashicorp.services/hashicorp/terraform:light
  steps:
    - attach_workspace:
        at: .
    - run:
        name: terraform destroy
        command: |
          terraform apply -auto-approve tfdestroy

You should not run these jobs unattended because it may lead to disruption to your services.

»Review the workflow

The last block in the configuration composes the jobs into a workflow. Workflows define order, precedence, and dependencies to perform the jobs within the pipeline.

workflows:
  version: 2
  plan_approve_apply:
    jobs:
      - plan-apply
      - hold-apply:
          type: approval
          requires:
            - plan-apply
      - apply:
          requires:
            - hold-apply
      - plan-destroy:
          requires:
            - apply
      - hold-destroy:
          type: approval
          requires:
            - plan-destroy
      - destroy:
          requires:
            - hold-destroy

workflows:
  version: 2
  plan_approve_apply:
    jobs:
      - plan-apply
      - hold-apply:
          type: approval
          requires:
            - plan-apply
      - apply:
          requires:
            - hold-apply
      - plan-destroy:
          requires:
            - apply
      - hold-destroy:
          type: approval
          requires:
            - plan-destroy
      - destroy:
          requires:
            - hold-destroy

Notice that hold-apply and hold-destroy have type: approval in this workflow. This means that for every run, CircleCI will generate a plan and wait for approval before running the apply job. CircleCI will also generate a plan to destroy the infrastructure and wait for approval before running the destroy job.

»Create Terraform Cloud workspace

This configuration uses Terraform Cloud for your project's state storage. You will create a new workspace to use for this project and configure it for local execution. When using local execution with Terraform Cloud, the Terraform operations occur in the environment that runs the Terraform CLI (in this case, the Docker executor configured for your build), and Terraform Cloud stores the state file for shared access across builds and runs.

In the Terraform Cloud UI, create a new CLI-driven workspace named learn-terraform-circleci.

On the workspace overview page, click on the curent Execution Mode to navigate to the general settings.

Under Execution Mode, select Local. Then, click Save Settings at the bottom of the page.

»Configure CircleCI project

Navigate to your CircleCI dashboard. Make sure that you are in the correct organization with access to your GitHub account by confirming the organization in the top left corner.

Then, select Projects in the left sidebar.

Search for your forked learn-terraform-circleci repository. Then, click the Set Up Project button

Select the Fastest configuration option to use the CircleCI configuration file in the repository. Enter the main branch as the brain to track. Then, click Set Up Project.

CircleCI will automatically attempt to run the job and fail because the project needs your AWS credentials and Terraform Cloud integration details.

Navigate to the project's Project Settings, then select Environment Variables from the sidebar.

Set the following environment variables, which Terraform will access in your build environment to configure both the AWS provider and the Terraform Cloud integration for your project:

Set AWS_ACCESS_KEY_ID to the generated key for the AWS user running this job. To generate an access key and secret access key file, log in to your AWS account and create them in IAM.
Set AWS_SECRET_ACCESS_KEY to the secret access key you generated above.
Set TF_CLOUD_ORGANIZATION to your Terraform Cloud organization name
Set TF_WORKSPACE to learn-terraform-circleci, the Terraform Cloud workspace you created and configured earlier in this tutorial.
Set TF_TOKEN_app_terraform_io to the API token you created at the beginning of the tutorial.

When complete, your environment variables page will list the configured variables.

»Define your Terraform variables

In your file editor, open terraform.tfvars and update the label variable value to hashicorp.fun. Then, save the file.

region = "us-east-1"
label  = "hashicorp.fun"
app    = "terramino"

terraform.tfvars

region = "us-east-1"
label  = "hashicorp.fun"
app    = "terramino"

»Trigger CircleCI workflow

Stage your changes to your GitHub repository.

$ git add terraform.tfvars

$ git add terraform.tfvars

Commit these changes with a message.

$ git commit -m "Update variable definitions"

$ git commit -m "Update variable definitions"

Finally, push these changes to your forked repository's main branch to kick off a CircleCI run.

$ git push

$ git push

The CircleCI web UI should indicate that your build started. The steps for this deployment will initialize your Terraform directory, plan the Terraform deployment, and wait for your approval to apply the planned changes.

Review the output for the plan-apply job, which shows the proposed execution plan to create your resources. First, click on the plan-apply step in the workflow, then expand the terraform init && plan step to review the execution plan.

Then, return to the workflow overview by clicking the plan_approve_apply workflow at the top and approve it.

Once the deployment job is complete, your workflow will hold so that you can review the destroy plan before it starts the destroy job. Before you approve the destroy operation, review the output for the apply job and navigate to the endpoint in the output to test out your application.

Any changes to your GitHub repository will trigger another run of this workflow. As with any Terraform deployment, Terraform will determine which resources to recreate or update in place depending on configuration and the provider.

Navigate to the Terraform Cloud workspace to verify that it stores the state for your project, and lists the resources it manages on the workspace overview page.

»Destroy the infrastructure

The plan-destroy step in the workflow generated and saved a plan to destroy your application. The hold-destroy job is a manual gate step that gives you to review the plan before the workflow destroys your resources. Click on the hold step and then choose "approve" to move on to the destroy job in this workflow.

While Terraform is destroying your infrastructure, navigate to the plan-destroy job in the CircleCI web UI to observe the output in the terraform create destroy plan dropdown.

»Next Steps

In this tutorial, you deployed an S3-backed website using Terraform and CircleCI and stored the project state in Terraform Cloud. If you would like to learn more about CircleCI or automated Terraform workflow best practices, consider these resources:

Infrastructure

Security

Applications

Networking

cloud