Deploy Terraform infrastructure with CircleCI

CircleCI is a continuous integration tool to automate software builds, tests, and deployments. The continuous integration workflow enables development teams to automate, self-test, quickly build, clone, and deploy software. Terraform deploys infrastructure repeatably. By adding Terraform into a CircleCI workflow, you can deploy your infrastructure alongside software in the same pipeline.

In this tutorial, you will use CircleCI to deploy an S3 backed web application. Your deployment process will be controlled by the CircleCI workflow to run your Terraform configuration in automation. CircleCI will be able to access your state file with an S3 remote backend for future deployments.

»Prerequisites

This tutorial assumes you have the following:

• A GitHub account
• A CircleCI account. Sign up with your GitHub account so CircleCI can build and deploy from your GitHub repositories. Review the CircleCI getting started guide for an introduction to the workflow if you are unfamiliar.
• An AWS account

Fork the demo code from GitHub. Once you have forked the repository, clone the demo code to your machine. You will need to edit the command below to include your GitHub username.

$ git clone https://github.com/YOUR-USER-NAME/learn-terraform-circleci

$ git clone https://github.com/YOUR-USER-NAME/learn-terraform-circleci

»Analyze a CircleCI configuration

Change directories to your forked repository.

$ cd learn-terraform-circleci/.circleci

$ cd learn-terraform-circleci/.circleci

Open the config.yml file in your file editor. This configuration defines the jobs for each step in your deployment.

There will be a total of four jobs in this workflow: plan-apply, apply, plan-destroy, and destroy. You will then use these jobs to define the automated Terraform workflow.

»Review the plan-apply job

Jobs are collections of steps CircleCI performs. Each job must declare an executor, an operating system which will launch and perform the actions you define, and a series of steps. You will use the docker executor in this tutorial to perform jobs to initialize, plan, apply, and destroy your Terraform configuration.

The plan job uses the hashicorp/terraform:light image which contains the Terraform binary. This job checkouts your repository, runs terraform init to initialize your configuration, and generates a plan file named tfapply by running terraform plan -out.

Finally, the persist_to_workspace step saves the initialized configuration and your environment variables for use in the following jobs. persist_to_workspace allows you to run this initialized Terraform configuration throughout the rest of the jobs in the workflow as if they are running on the same machine.

version: 2

jobs:
  plan-apply:
    working_directory: /tmp/project
    docker:
      - image: docker.mirror.hashicorp.services/hashicorp/terraform:light
    steps:
      - checkout
      - run:
          name: terraform init & plan
          command: |
            terraform init -input=false
            terraform plan -out tfapply -var-file variables.tfvars
      - persist_to_workspace:
          root: .
          paths:
            - .

version: 2
jobs:  plan-apply:    working_directory: /tmp/project    docker:      - image: docker.mirror.hashicorp.services/hashicorp/terraform:light    steps:      - checkout      - run:          name: terraform init & plan          command: |            terraform init -input=false            terraform plan -out tfapply -var-file variables.tfvars      - persist_to_workspace:          root: .          paths:            - .

The steps in this configuration are actions that CircleCI takes in the workflow to perform your job. Steps are a collection of executable commands. For example, the checkout step checks out the source code for a job over SSH. The steps in this job run CLI commands terraform init and terraform plan -out with your defined variables.

»Review the apply job

The next job in this config is the apply job. The attach_workspace step in the apply job loads the previously persisted workspace in the plan job. The apply job runs terraform apply using the execution plan generated from the previous job.

apply:
  docker:
    - image: docker.mirror.hashicorp.services/hashicorp/terraform:light
  steps:
    - attach_workspace:
        at: .
    - run:
        name: terraform
        command: |
          terraform apply -auto-approve tfapply
    - persist_to_workspace:
        root: .
        paths:
          - .

apply:  docker:    - image: docker.mirror.hashicorp.services/hashicorp/terraform:light  steps:    - attach_workspace:        at: .    - run:        name: terraform        command: |          terraform apply -auto-approve tfapply    - persist_to_workspace:        root: .        paths:          - .

»Review the plan-destroy and destroy jobs

The plan-destroy job creates an execution plan and the destroy job executes that with terraform apply tfdestroy job to remove all of the infrastructure you created.

plan-destroy:
  docker:
    - image: docker.mirror.hashicorp.services/hashicorp/terraform:light
  steps:
    - attach_workspace:
        at: .
    - run:
        name: terraform create destroy plan
        command: |
          terraform plan -destroy -out tfdestroy -var-file variables.tfvars
    - persist_to_workspace:
        root: .
        paths:
          - .

destroy:
  docker:
    - image: docker.mirror.hashicorp.services/hashicorp/terraform:light
  steps:
    - attach_workspace:
        at: .
    - run:
        name: terraform destroy
        command: |
          terraform apply -auto-approve tfdestroy

plan-destroy:  docker:    - image: docker.mirror.hashicorp.services/hashicorp/terraform:light  steps:    - attach_workspace:        at: .    - run:        name: terraform create destroy plan        command: |          terraform plan -destroy -out tfdestroy -var-file variables.tfvars    - persist_to_workspace:        root: .        paths:          - .
destroy:  docker:    - image: docker.mirror.hashicorp.services/hashicorp/terraform:light  steps:    - attach_workspace:        at: .    - run:        name: terraform destroy        command: |          terraform apply -auto-approve tfdestroy

You should not run these jobs unattended because it may lead to disruptions in your infrastructure. In the next section, your CircleCI configuration defines the order of operations for these jobs in a workflow.

»Review the workflow

Finally, the last block in the configuration is the workflows. Workflow defines order, precedence, and requirements to perform the jobs within the pipeline. This workflow runs the jobs detailed throughout the config and details the order in which those jobs run as well as the dependencies of each job. The plan_approve_apply workflow runs through all of the steps in order and creates the application.

workflows:
  version: 2
  plan_approve_apply:
    jobs:
      - plan-apply
      - hold-apply:
          type: approval
          requires:
            - plan-apply
      - apply:
          requires:
            - hold-apply
      - plan-destroy:
          requires:
            - apply
      - hold-destroy:
          type: approval
          requires:
            - plan-destroy
      - destroy:
          requires:
            - hold-destroy

workflows:  version: 2  plan_approve_apply:    jobs:      - plan-apply      - hold-apply:          type: approval          requires:            - plan-apply      - apply:          requires:            - hold-apply      - plan-destroy:          requires:            - apply      - hold-destroy:          type: approval          requires:            - plan-destroy      - destroy:          requires:            - hold-destroy

Notice that hold-apply and hold-destroy have type: approval in this workflow. This means that for every run, CircleCI will generate a plan and wait for approval before running the apply job. CircleCI will also generate a plan to destroy the infrastructure and wait for approval before running the destroy job.

»Setup the CircleCI UI

In order to pass this configuration to the CircleCI jobs, you have to setup your CirleCI project.

In the CircleCI web UI, choose the Projects icon from the left. Search for the repo you forked and choose Set Up Project.

Choose "Hello World" as the language and choose the "Use Existing Config" option.

Choose "Start Building" and you should be presented with a popup confirming you have created config.yml file.

CircleCI will automatically attempt to run the job and fail because the project needs your AWS credentials.

In the right-hand corner of this page, go the Project Settings and navigate to Environment Variables.

The two environment variables for this project are AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY. These are variables that CircleCI uses to inject data into the .config.yml file.

1. AWS_ACCESS_KEY_ID is the generated keys for the AWS user running this job. To generate an access key and secret access key file, log in to your AWS account and create them in IAM.
2. AWS_SECRET_ACCESS_KEY is bundled with the access key above.

Add and save these variables in your CircleCI Build Settings.

»Create a remote backend

You need to save the Terraform state file for future runs. In this workflow, your remote state storage will be in an S3 bucket backend.

In your terminal, change into the s3_backend directory of the learn-terraform-circleci repository.

$ cd s3_backend

$ cd s3_backend

Initialize your configuration.

$ terraform init

$ terraform init

Then, apply your configuration to create an S3 bucket. Respond to the confirmation prompt with a yes.

$ terraform apply
Terraform used the selected providers to generate the following execution plan. Resource actions are indicated with the following symbols:
  + create

Terraform will perform the following actions:

  # aws_s3_bucket.terraform_state will be created
  ## ...

Plan: 1 to add, 0 to change, 0 to destroy.

Changes to Outputs:
  + s3_bucket_name = (known after apply)

Do you want to perform these actions?
  Terraform will perform the actions described above.
  Only 'yes' will be accepted to approve.

  Enter a value: yes

$ terraform applyTerraform used the selected providers to generate the following execution plan. Resource actions are indicated with the following symbols:  + create
Terraform will perform the following actions:
  # aws_s3_bucket.terraform_state will be created  ## ...
Plan: 1 to add, 0 to change, 0 to destroy.
Changes to Outputs:  + s3_bucket_name = (known after apply)
Do you want to perform these actions?  Terraform will perform the actions described above.  Only 'yes' will be accepted to approve.
  Enter a value: yes

Save the output of the random bucket name. This is the encrypted bucket for your remote state storage.

Apply complete! Resources: 1 added, 0 changed, 0 destroyed.

Outputs:

s3_bucket_name = "circle-ci-backend-20210609001340736000000001"

Apply complete! Resources: 1 added, 0 changed, 0 destroyed.
Outputs:
s3_bucket_name = "circle-ci-backend-20210609001340736000000001"

Change into the root directory of the learn-terraform-circleci local repository.

$ cd ..

$ cd ..

Open the main.tf file and add the backend configuration with your unique bucket ID and region.

terraform {
  required_providers {
    aws = {
      source = "hashicorp/aws"
    }
  }
+  backend "s3" {
+    bucket = "YOUR-UNIQUE-BUCKET-ID"
+    key    = "terraform/webapp/terraform.tfstate"
+    region = "us-east-1"
+ }
}

terraform {  required_providers {    aws = {      source = "hashicorp/aws"    }  }+  backend "s3" {+    bucket = "YOUR-UNIQUE-BUCKET-ID"+    key    = "terraform/webapp/terraform.tfstate"+    region = "us-east-1"+ }}

Save this file.

»Define your Terraform variables

In your file editor, make an edit to the variables.tfvars file to define variables for your deployment.

region = "us-east-1"
label  = "hashicorp.fun"
app    = "terramino"
user   = "circleci-user"

region = "us-east-1"label  = "hashicorp.fun"app    = "terramino"user   = "circleci-user"

Replace the region variable with an AWS region closest to you and save the file.

»Trigger CircleCI workflow

Prepare to add your changes to your GitHub repository.

$ git add main.tf variables.tfvars

$ git add main.tf variables.tfvars

Commit these changes with a message.

$ git commit -m "Add remote backend and variable definitions"

$ git commit -m "Add remote backend and variable definitions"

Finally, push these changes to your forked repositories master branch to kick off a CircleCI run.

$ git push

$ git push

The CircleCI web UI should indicate that your build has started. The steps for this deployment will initialize your Terraform directory, plan the Terraform deployment, and wait for your approval to apply the planned changes.

Approve the apply step.

Review that job by navigating to the workflows for plan_approve_apply. In each job in the workflow, you can click on each step to expand the output.

Once the deployment job is complete, your workflow will be on hold. The hold in your configuration is the step before destruction. The output in the job displays your webapp address. Navigate to that address and verify your app deployed correctly.

Any changes to your GitHub repo will trigger another run of this workflow. As with any Terraform deployment, some resources will be destroyed or updated in place depending on the configuration and the cloud provider you are using in the future.

»Generate the destroy plan

The plan-destroy step in the workflow will prevent CircleCI from continuing to the next job in your workflow. hold-destroy is a manual gate step which allows you to decide when to move to the final step in the configuration. Click on the hold step and then choose "approve" to move on to the destroy job in this workflow.

Approve the destroy job.

Navigate to the plan-destroy job in the CircleCI web UI to observe the output in the "terraform destroy" dropdown.

»Destroy your state S3 bucket

When you have completed this tutorial, you should destroy the S3 bucket you created for your state file.

In your local repository, navigate to the s3_bucket directory.

$ cd s3_bucket

$ cd s3_bucket

Run terraform destroy to destroy the S3 bucket.

$ terraform destroy
aws_s3_bucket.terraform_state: Refreshing state... [id=circle-ci-backend-20210609001340736000000001]

## ...

Plan: 0 to add, 0 to change, 1 to destroy.

Changes to Outputs:
  - s3_bucket_name = "circle-ci-backend-20210609001340736000000001" -> null

Do you really want to destroy all resources?
  Terraform will destroy all your managed infrastructure, as shown above.
  There is no undo. Only 'yes' will be accepted to confirm.

  Enter a value: yes

aws_s3_bucket.terraform_state: Destroying... [id=circle-ci-backend-20210609001340736000000001]
aws_s3_bucket.terraform_state: Destruction complete after 1s

Destroy complete! Resources: 1 destroyed.

$ terraform destroyaws_s3_bucket.terraform_state: Refreshing state... [id=circle-ci-backend-20210609001340736000000001]
## ...
Plan: 0 to add, 0 to change, 1 to destroy.
Changes to Outputs:  - s3_bucket_name = "circle-ci-backend-20210609001340736000000001" -> null
Do you really want to destroy all resources?  Terraform will destroy all your managed infrastructure, as shown above.  There is no undo. Only 'yes' will be accepted to confirm.
  Enter a value: yes
aws_s3_bucket.terraform_state: Destroying... [id=circle-ci-backend-20210609001340736000000001]aws_s3_bucket.terraform_state: Destruction complete after 1s
Destroy complete! Resources: 1 destroyed.

Type yes to approve the destroy job.

»Next Steps

In this tutorial, you deployed a hosted webpage in S3 by integrating Terraform with your CircleCI workflow. If you would like to learn more about CircleCI or automated Terraform workflow best practices, consider these resources:

• Deploy your Terraform infrastructure with the official CircleCI Terraform Orb
• Running Terraform in Automation
• CircleCI Configuration reference
• Automate Your Infrastructure with GitOps, Terraform & CircleCI