Virtual Event
Join us for the next HashiConf Digital October 12-15, 2020 Register for Free

Call APIs with Terraform Providers

Perform CRUD operations with Providers

In this tutorial, you use a Terraform provider to interact with a fictional coffee-shop application, HashiCups. In the process, you will learn how providers map target APIs to Terraform in order to create, read, update and delete resources.

Later in the track, you will re-create the HashiCups provider discussed in this tutorial based on the Terraform Plugin SDK v2.

»Terraform plugins

Terraform is comprised of Terraform Core and Terraform Plugins.

core-plugins-api

  1. Terraform Core reads the configuration and builds the resource dependency graph.
  2. Terraform Plugins (providers and provisioners) bridge Terraform Core and their respective target APIs. Terraform provider plugins implement resources via basic CRUD (create, read, update, and delete) APIs to communicate with third party services.

Upon terraform plan or terraform apply, Terraform Core asks the Terraform provider to perform an action via a RPC interface. The provider attempts to fulfill the request by invoking a CRUD operation against the target API's client library. This process enforces a clear separation of concerns. Providers are able to serve as an abstraction of a client library.

While most Terraform providers manage cloud infrastructure (e.g. AWS, Azure and GCP providers), providers can serve as an interface to any API and allow Terraform to potentially manage any resource.

»Introduction to HashiCups

HashiCups is a demo application that allows you to view and order customized HashiCorp branded coffee.

The API has a number of unprotected and protected endpoints. You can list all coffees and ingredients for a particular coffee without authentication. Once authenticated, you can create, read, update and delete (CRUD) orders.

The Terraform HashiCups provider interfaces with the product's REST API through a GoLang client library. This allows you to manage HashiCups orders through Terraform.

»Prerequisites

To follow this tutorial, you need:

It also assumes that you are familiar with the usual Terraform plan/apply workflow. If you're new to Terraform itself, refer first to the Getting Started track.

»Initialize HashiCups locally

The HashiCups provider requires a running instance of HashiCups. In your terminal, clone the Learn HashiCups Provider repository and navigate to it.

$ git clone https://github.com/hashicorp/learn-terraform-hashicups-provider && cd learn-terraform-hashicups-provider

Navigate to the docker_compose directory and run docker-compose up to spin up a local instance of HashiCups on port :19090.

$ cd docker_compose && docker-compose up

Leave this process running in your terminal window. This will contain HashiCups' log messages.

In another terminal window, verify that HashiCups is running by sending a request to its health check endpoint.

$ curl localhost:19090/health
ok

»Create new HashiCups user

HashiCups requires a username and password to generate an JSON web token (JWT) which is used to authenticate against protected endpoints. You will use this user to authenticate to the HashiCups provider to manage your orders.

Create a user on HashiCups named education with the password test123.

$ curl -X POST localhost:19090/signup -d '{"username":"education", "password":"test123"}'
{"UserID":1,"Username":"education","token":"eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJleHAiOjE1OTEwNzgwODUsInVzZXJfaWQiOjIsInVzZXJuYW1lIjoiZWR1Y2F0aW9uIn0.CguceCNILKdjOQ7Gx0u4UAMlOTaH3Dw-fsll2iXDrYU"}

Then, authenticate to HashiCups. This will return the userID, username, and a JWT token. Your JWT authorization token will be used later to retrieve your orders.

$ curl -X POST localhost:19090/signin -d '{"username":"education", "password":"test123"}'
{"UserID":1,"Username":"education","token":"eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJleHAiOjE1OTEwNzgwODUsInVzZXJfaWQiOjIsInVzZXJuYW1lIjoiZWR1Y2F0aW9uIn0.CguceCNILKdjOQ7Gx0u4UAMlOTaH3Dw-fsll2iXDrYU"}

Set HASHICUPS_TOKEN to the token you retrieved from invoking the /signin endpoint. You will use this later in the tutorial to verify your HashiCups order has been created, updated and deleted.

$ export HASHICUPS_TOKEN=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJleHAiOjE1OTEwNzgwODUsInVzZXJfaWQiOjIsInVzZXJuYW1lIjoiZWR1Y2F0aW9uIn0.CguceCNILKdjOQ7Gx0u4UAMlOTaH3Dw-fsll2iXDrYU

The terminal containing your HashiCups logs will have recorded both operations.

api_1  | 2020-07-16T09:19:50.601Z [INFO]  Handle User | signup
api_1  | 2020-07-16T09:19:59.601Z [INFO]  Handle User | signin

Now that the HashiCups app is running, you're ready to interact with it using the Terraform provider.

»Install HashiCups provider

With Terraform 0.13, you must specify all required providers and their respective source in your Terraform configuration. A provider source string is comprised of [hostname]/[namespace]/[name].

  • Both hostname and namespace are optional for providers in the hashicorp namespace in the Terraform Registry.
  • If hostname is omitted, Terraform will use the Terraform Registry hostname as the default hostname.
  • namespace is always required when hostname is set.

For third party Terraform providers, you can choose any arbitrary identifier (comprised of letters and hyphens) for the hostname and namespace.

When you run terraform init, Terraform will attempt to locate the provider locally in the appropriate subdirectory within the user plugins directory, ~/.terraform.d/plugins/${name}/${version}/${os}_${arch} or %APPDATA%\terraform.d\plugins\${name}\${version}\${os}_${arch}. If it isn't located and it is a partner/community Terraform provider, it will attempt to download the provider from the Terraform Registry.

In order to use any third-party provider not published to the registry, you must download the binary then move it into appropriate subdirectory within the user plugins directory. Because HashiCups is a third-party provider used primarily for demos and education, you will do this. The provider source string for the HashiCups provider will be hashicorp.com/edu/hashicups.

First, download the HashiCups provider binary. Because providers are Go binaries, you can download a pre-compiled binary or build it directly from source. Building from source requires some Go experience.

Download the HashiCups provider v0.2 binary that matches your system. A full list of binaries can be found on the Terraform HashiCups Provider release page. Rename the file to terraform-provider-hashicups.

Download the HashiCups provider for MacOS.

$ curl -Lo terraform-provider-hashicups https://github.com/hashicorp/terraform-provider-hashicups/releases/download/v0.2/terraform-provider-hashicups_0.2_darwin_amd64

Then, make it an executable.

$ chmod +x terraform-provider-hashicups

Create the appropriate subdirectory within the user plugins directory for the HashiCups provider.

$ mkdir -p ~/.terraform.d/plugins/hashicorp.com/edu/hashicups/0.2/darwin_amd64

Finally, move the HashiCups provider binary into the newly created directory.

$ mv terraform-provider-hashicups ~/.terraform.d/plugins/hashicorp.com/edu/hashicups/0.2/darwin_amd64

Now that the provider is in your user plugins directory, you can use the provider in your Terraform configuration.

»Initialize workspace

Add the following to your main.tf file. This is required for Terraform 0.13+.

terraform {
  required_providers {
    hashicups = {
      versions = ["0.2"]
      source = "hashicorp.com/edu/hashicups"
    }
  }
}

Since HashiCups is a third-party provider, the hostname and namespace values in the source string are arbitrary. For more information on provider source, reference the provider source documentation.

Then, initialize your Terraform workspace. If your Hashicups provider is located in the correct directory, it should successfully initialize. Otherwise, move your HashiCups provider to the correct directory: ~/.terraform.d/plugins/${name}/${version}/${os}_${arch}.

$ terraform init

»Create order

Add the following to your main.tf file.

This authenticate the HashiCups provider, create an order and return the order's values in your output. The order contains total of 4 coffees: 2 of each coffee_id 3 and 2.

provider "hashicups" {
  username = "education"
  password = "test123"
}

resource "hashicups_order" "edu" {
  items {
    coffee {
      id = 3
    }
    quantity = 2
  }
  items {
    coffee {
      id = 2
    }
    quantity = 2
  }
}

output "edu_order" {
  value = hashicups_order.edu
}

Run terraform apply to create the order. Notice how the execution plan shows a proposed order, with additional information about the order. Remember to confirm the apply step with a yes.

$ terraform apply

An execution plan has been generated and is shown below.
Resource actions are indicated with the following symbols:
  + create

Terraform will perform the following actions:

  # hashicups_order.edu will be created
  + resource "hashicups_order" "edu" {
      + id           = (known after apply)
      + last_updated = (known after apply)

      + items {
          + quantity = 2

          + coffee {
              + description = (known after apply)
              + id          = 3
              + image       = (known after apply)
              + name        = (known after apply)
              + price       = (known after apply)
              + teaser      = (known after apply)
            }
        }
      + items {
          + quantity = 2

          + coffee {
              + description = (known after apply)
              + id          = 2
              + image       = (known after apply)
              + name        = (known after apply)
              + price       = (known after apply)
              + teaser      = (known after apply)
            }
        }
    }

Plan: 1 to add, 0 to change, 0 to destroy.

Once the apply completes, the provider saves the resource's state. If you're running Terraform locally, your terraform.tfstate file contains this state. You can also view the state by running terraform state show <resource_name>.

$ terraform state show hashicups_order.edu
# hashicups_order.edu:
resource "hashicups_order" "edu" {
  id = "1"

  items {
    quantity = 2

    coffee {
      id     = 3
      image  = "/nomad.png"
      name   = "Nomadicano"
      price  = 150
      teaser = "Drink one today and you will want to schedule another"
    }
  }
  items {
    quantity = 2

    coffee {
      id     = 2
      image  = "/vault.png"
      name   = "Vaulatte"
      price  = 200
      teaser = "Nothing gives you a safe and secure feeling like a Vaulatte"
    }
  }
}

The (known after apply) values in the execution plan during the terraform apply state have all been populated, since the order was successfully created.

»Verify order created

When you created an order in HashiCups using Terraform, the terminal containing your HashiCups logs will have recorded operations invoked by the HashiCups provider.

api_1  | 2020-07-16T10:47:47.838Z [INFO]  Handle User | signin
api_1  | 2020-07-16T10:47:54.841Z [INFO]  Handle User | signin
api_1  | 2020-07-16T10:47:54.853Z [INFO]  Handle Orders | CreateOrder
api_1  | 2020-07-16T10:47:54.866Z [INFO]  Handle Orders | GetUserOrder

The provider invoked a total of 4 operations.

  1. The provider invoked the first signin operation when you ran terraform apply to retrieve the current state of the resources. Because there are no resources, it only authenticated the user.
  2. The provider invoked the second signin operation after you confirmed the apply run. The provider authenticated using the provided credentials to retrieve and save the JWT token.
  3. The provider invoked the CreateOrder operation to create the order defined by the Terraform configuration. Since this is a protected endpoint, it used the saved JWT token from the signin operation.
  4. After the order was created, the provider invoked the GetUserOrder operation to retrieve the order detail. Since this is a protected endpoint, it used the saved JWT token from the signin operation.

Verify the order was created by retrieving the order details via the API.

$ curl -X GET  -H "Authorization: ${HASHICUPS_TOKEN}" localhost:19090/orders/1
{"id":1,"items":[{"coffee":{"id":3,"name":"Nomadicano","teaser":"Drink one today and you will want to schedule another","description":"","price":150,"image":"/nomad.png","ingredients":null},"quantity":2},{"coffee":{"id":2,"name":"Vaulatte","teaser":"Nothing gives you a safe and secure feeling like a Vaulatte","description":"","price":200,"image":"/vault.png","ingredients":null},"quantity":2}]}

The order's properties should be the same as that of your hashicups_order.edu resource.

»Update order

Now, change your order by updating the order resource through Terraform. In your main.tf, update the coffee quantity in hashicups_order.edu block.

resource "hashicups_order" "edu" {
  items {
    coffee {
      id = 3
    }
-    quantity = 2
+    quantity = 3
  }
  items {
    coffee {
      id = 2
    }
-    quantity = 2
+    quantity = 1
  }
}

Run terraform apply to update the order. Notice how the execution plan reflects the order change. Terraform is able to determine that these changes could be done in place rather than destroying the current order and creating a new one. This is because the modified property (quantity) does not have ForceNew set to true.

Remember to confirm the apply step with a yes.

$ terraform apply
hashicups_order.edu: Refreshing state... [id=7]

An execution plan has been generated and is shown below.
Resource actions are indicated with the following symbols:
  ~ update in-place

Terraform will perform the following actions:

  # hashicups_order.edu will be updated in-place
  ~ resource "hashicups_order" "edu" {
        id = "1"

      ~ items {
          ~ quantity = 2 -> 3

            coffee {
                id     = 3
                image  = "/nomad.png"
                name   = "Nomadicano"
                price  = 150
                teaser = "Drink one today and you will want to schedule another"
            }
        }
      ~ items {
          ~ quantity = 2 -> 1

            coffee {
                id     = 2
                image  = "/vault.png"
                name   = "Vaulatte"
                price  = 200
                teaser = "Nothing gives you a safe and secure feeling like a Vaulatte"
            }
        }
    }

Plan: 0 to add, 1 to change, 0 to destroy.

Once the apply completes, view the output. Notice that while the order_id remained the same (1), the provider updated the order's coffee quantity and assigned the current time to the newly created last_updated field.

Outputs:

edu_order = {
  "id" = "1"
  "items" = [
    {
      "coffee" = [
        {
          "description" = ""
          "id" = 3
          "image" = "/nomad.png"
          "name" = "Nomadicano"
          "price" = 150
          "teaser" = "Drink one today and you will want to schedule another"
        },
      ]
      "quantity" = 3
    },
    {
      "coffee" = [
        {
          "description" = ""
          "id" = 2
          "image" = "/vault.png"
          "name" = "Vaulatte"
          "price" = 200
          "teaser" = "Nothing gives you a safe and secure feeling like a Vaulatte"
        },
      ]
      "quantity" = 1
    },
  ]
  "last_updated" = "Thursday, 16-Jul-20 04:13:48 PDT"
}

»Verify order updated

Check the terminal containing your HashiCups logs for the recorded operations invoked by the HashiCups provider.

api_1  | 2020-07-16T11:13:14.964Z [INFO]  Handle User | signin
api_1  | 2020-07-16T11:13:14.973Z [INFO]  Handle Orders | GetUserOrder
api_1  | 2020-07-16T11:13:15.084Z [INFO]  Handle User | signin
api_1  | 2020-07-16T11:13:48.400Z [INFO]  Handle User | signin
api_1  | 2020-07-16T11:13:48.416Z [INFO]  Handle Orders | UpdateOrder
api_1  | 2020-07-16T11:13:48.421Z [INFO]  Handle Orders | GetUserOrder

The provider invoked a total of 6 operations.

  1. The provider invoked the first signin operation when you ran terraform apply to retrieve the current state of the resources. Because there are resources in the state...
  2. The provider invoked the GetUserOrder operation to reconcile any potential differences between the current state (state in HashiCups) and the state stored by Terraform.
  3. The provider invoked the second signin operation after Terraform's state was updated.
  4. The provider invoked the third signin operation after you confirmed the apply run. The provider authenticated using the provided credentials to retrieve and save the JWT token.
  5. The provider invoked the UpdateOrder operation to update the order to the one defined by the Terraform configuration. Since this is a protected endpoint, it used the saved JWT token from the signin operation.
  6. After the order was updated, the provider invoked the GetUserOrder operation to retrieve the order detail. Since this is a protected endpoint, it used the saved JWT token from the signin operation.

Verify the order was updated by retrieving the order details via the API. Substitute the TOKEN with the JWT token you received when you authenticated to HashiCups.

$ curl -X GET  -H "Authorization: ${HASHICUPS_TOKEN}" localhost:19090/orders/1
{"id":1,"items":[{"coffee":{"id":3,"name":"Nomadicano","teaser":"Drink one today and you will want to schedule another","description":"","price":150,"image":"/nomad.png","ingredients":null},"quantity":3},{"coffee":{"id":2,"name":"Vaulatte","teaser":"Nothing gives you a safe and secure feeling like a Vaulatte","description":"","price":200,"image":"/vault.png","ingredients":null},"quantity":1}]}

The order's properties should be the same as that of your updated hashicups_order.edu resource. There should be 3 Nomadicano and 1 Vaulatte.

»Read coffee ingredients

Add the following data blocks to your main.tf file. This will retrieve ingredients for the first coffee from your order. The data block retrieves additional information about a resource, which enables it to be referenced by another Terraform resource. In this example, it's used by an output block to display the coffee ingredients.

data "hashicups_ingredients" "first_coffee" {
  coffee_id = hashicups_order.edu.items[0].coffee[0].id
}

output "first_coffee_ingredients" {
  value = data.hashicups_ingredients.first_coffee
}

Run terraform apply to retrieve the ingredients. Remember to confirm the apply step with a yes.

$ terraform apply
hashicups_order.edu: Refreshing state... [id=7]
data.hashicups_ingredients.first_coffee: Refreshing state...

Apply complete! Resources: 0 added, 0 changed, 0 destroyed.

Outputs:

first_coffee_ingredients = {
  "coffee_id" = 3
  "id" = "3"
  "ingredients" = [
    {
      "id" = 1
      "name" = "ingredient - Espresso"
      "quantity" = 20
      "unit" = "ml"
    },
    {
      "id" = 3
      "name" = "ingredient - Hot Water"
      "quantity" = 100
      "unit" = "ml"
    },
  ]
}

As you can see, ingredients of a Nomadicano is 20 ml espresso and 100 ml hot water.

»Verify operation

Check the terminal containing your HashiCups logs for the recorded operations invoked by the HashiCups provider.

api_1  | 2020-07-16T11:36:17.907Z [INFO]  Handle User | signin
api_1  | 2020-07-16T11:36:17.915Z [INFO]  Handle Orders | GetUserOrder
api_1  | 2020-07-16T11:36:17.921Z [INFO]  Handle Coffee Ingredients
api_1  | 2020-07-16T11:36:18.035Z [INFO]  Handle User | signin
api_1  | 2020-07-16T11:36:27.513Z [INFO]  Handle User | signin

After the provider reconciles the state (first two operations), it invokes Handle Coffee Ingredients which retrieves the coffee ingredients.

»Delete order

Finally, you can delete resources in Terraform. Run terraform destroy. Remember to confirm the destroy step with a yes.

$ terraform destroy
hashicups_order.edu: Refreshing state... [id=7]
data.hashicups_ingredients.first_coffee: Refreshing state...

An execution plan has been generated and is shown below.
Resource actions are indicated with the following symbols:
  - destroy

Terraform will perform the following actions:

  # hashicups_order.edu will be destroyed
  - resource "hashicups_order" "edu" {
      - id           = "1" -> null
      - last_updated = "Thursday, 16-Jul-20 04:13:48 PDT"

      - items {
          - quantity = 3 -> null

          - coffee {
              - id     = 3 -> null
              - image  = "/nomad.png" -> null
              - name   = "Nomadicano" -> null
              - price  = 150 -> null
              - teaser = "Drink one today and you will want to schedule another" -> null
            }
        }
      - items {
          - quantity = 1 -> null

          - coffee {
              - id     = 2 -> null
              - image  = "/vault.png" -> null
              - name   = "Vaulatte" -> null
              - price  = 200 -> null
              - teaser = "Nothing gives you a safe and secure feeling like a Vaulatte" -> null
            }
        }
    }

Plan: 0 to add, 0 to change, 1 to destroy.

When the destroy step completes, the provider has destroyed the order resource, reflected by an empty state file.

»Verify order deleted

Check the terminal containing your HashiCups logs for the recorded operations invoked by the HashiCups provider.

api_1  | 2020-07-16T11:42:59.915Z [INFO]  Handle User | signin
api_1  | 2020-07-16T11:42:59.924Z [INFO]  Handle Orders | GetUserOrder
api_1  | 2020-07-16T11:42:59.929Z [INFO]  Handle Coffee Ingredients
api_1  | 2020-07-16T11:43:01.683Z [INFO]  Handle User | signin
api_1  | 2020-07-16T11:43:01.689Z [INFO]  Handle Orders | DeleteOrder

After the provider reconciles the state (first three operations), it destroys the order.

Verify the order was deleted by retrieving the order details via the API. Substitute the TOKEN with the JWT token you received when you authenticated to HashiCups.

$ curl -X GET  -H "Authorization: ${HASHICUPS_TOKEN}" localhost:19090/orders/1
{}

The order is blank, as expected.

»Next steps

In this tutorial, you created, read, updated and deleted an order resource in HashiCorp's demo application. In the process, you've learned how Terraform interacts with target APIs via providers.

A full list of official, partner and community Terraform providers can be found on the Terraform Provider Registry. We encourage you to find provider you're interested in and experiment!

terraform-registry-walkthrough

To learn more about provider source, refer to Automatic Installation of Third-Party Providers with Terraform 0.13 blog post.

To learn more about interpolation, refer to the Configuration Language documentation.

In the following tutorials, you will learn how to write a Terraform provider using the Plugins SDK v2 by re-implementing the HashiCups provider. In the process, you will create data sources, authenticate the provider to the HashiCups client, and create resources with CRUD functionality.