Maintaining Multiple Environments with Terraform

Separating concerns between different environments presents a new set of challenges. If a change occurs in development, the user needs to ensure no unnecessary changes will effect production. In order to segment our infrastructure, let's take a look at some best practices.

» Remote State

The getting started guide introduced the concept of remote backends with Consul.

Terraform stores state locally in a file named terraform.tfstate. Local state files complicate team usage of Terraform so you can use remote state to ensure that all team members can pull from the same data source and share that state file. The example usage in this guide will be with HashiCorp Consul, but Terraform supports storing state in Terraform Enterprise, Amazon S3, Microsoft Azure Storage, and more.

» How to Store State Remotely

We'll use the demo Consul server for this guide. This should not be used for real data. Additionally, the demo server doesn't permit locking. If you want to play with state locking, you'll have to run your own Consul server or use a backend that supports locking.

First, configure the backend in your configuration:

terraform {
  backend "consul" {
    address = "demo.consul.io"
    path    = "getting-started-RANDOMSTRING"
    lock    = false

Please replace "RANDOMSTRING" with some random text. The demo server is public and we want to try to avoid overlapping with someone else running through the getting started guide.

The backend section configures the backend you want to use. After configuring a backend, run terraform init to setup Terraform. It should ask if you want to migrate your state to Consul. Say "yes" and Terraform will copy your state.

Now, if you run terraform apply, Terraform should state that there are no changes:

$ terraform apply
# ...

No changes. Infrastructure is up-to-date.

This means that Terraform did not detect any differences between your
configuration and real physical resources that exist. As a result, Terraform
doesn't need to do anything.

Terraform is now storing your state remotely in Consul. Remote state storage makes collaboration easier and keeps state and secret information off your local disk. Remote state is loaded only in memory when it is used.

If you want to move back to local state, you can remove the backend configuration block from your configuration and run terraform init again. Terraform will once again ask if you want to migrate your state back to local.

» Workspaces

Workspaces ensure that each Terraform configuration maintains a separate backend for persistent data in the Terraform state. In a single Terraform configuration, the "default" workspace holds the only state file associated with that configuration.

» How to use Multiple Workspaces

We can begin by separating our Terraform configurations with the workspace command.

$ terraform workspace new prod
Created and switched to workspace "prod"!

You're now on a new, empty workspace. Workspaces isolate their state,
so if you run "terraform plan" Terraform will not see any existing state
for this configuration.

$ terraform workspace new dev
Created and switched to workspace "dev"!

You're now on a new, empty workspace. Workspaces isolate their state,
so if you run "terraform plan" Terraform will not see any existing state
for this configuration.

As the command says, if you run terraform plan, Terraform will not see any existing resources that existed on the default (or any other) workspace. These resources still physically exist, but are managed in another Terraform workspace. Running terraform workspace show will show which workspace you are currently working with.

You can also reference the current workspace in your configuration using interpolation formed as ${terraform.workspace} and can be useful for changing behavior based on that workspace.

» When to use Multiple Workspaces

Workspaces alone are not a suitable tool for system decomposition, because each subsystem should have its own separate configuration and backend, and will thus have its own distinct set of workspaces. When utilizing remote backends with workspaces, you have strong separation between multiple deployments.

Multiple workspaces are also particularly useful for feature branches in version control. The default workspace might correspond to the "master" or "trunk" branch, which describes the intended state of production infrastructure. When a feature branch is created to develop a change, the developer of that feature might create a corresponding workspace and deploy into it a temporary "copy" of the main infrastructure so that changes can be tested without affecting the production infrastructure. Once the change is merged and deployed to the default workspace, the test infrastructure can be destroyed and the temporary workspace deleted.

» Sensitive Data

When using remote state, state is only ever held in memory when used by Terraform. It may be encrypted at rest but this depends on the specific remote state backend.

It is important to keep this in mind if you do (or plan to) store sensitive data (e.g. database passwords, user passwords, private keys) as it may affect the risk of exposure of such sensitive data.

» Recommendations

Terraform state can contain sensitive data depending on the resources in-use and your definition of "sensitive." The state contains resource IDs and all resource attributes. For resources such as databases, this may contain initial passwords.

Some resources (such as RDS databases) have options for PGP encrypting the values within the state. This is implemented on a per-resource basis and you should assume the value is plaintext unless otherwise documented.

When using local state, state is stored in plain-text JSON files. Storing state remotely may provide you encryption at rest depending on the backend you choose. As of Terraform 0.9, Terraform will only hold the state value in memory when remote state is in use. It is never explicitly persisted to disk.

For example, encryption at rest can be enabled with the S3 backend and IAM policies and logging can be used to identify any invalid access. Requests for the state go over a TLS connection.

Terraform Enterprise is a commercial product from HashiCorp that also acts as a backend and provides encryption at rest for state. Terraform Enterprise also knows the identity of the user requesting state and maintains a history of state changes. This can be used to provide access control and detect any breaches.

» Contact Us

For any questions or feedback please contact us at terraform-provider-dev@hashicorp.com.