Use Application Load Balancers for Blue-Green and Canary Deployments

Blue-green deployments and rolling deployments (canary tests) allow you to release new software gradually, and identify and mitigate the potential blast radius of a failed software release. This allows you to release new software with near-zero downtime.

In a blue-green deployment, the current service acts as the blue environment. When a new version of the service is available, you deploy the new service and underlying infrastructure into a new green environment. When the green environment is ready and tested, traffic is redirected from the blue environment to the green one. This process ensures that:

Errors are identified when the green environment is tested. If there are any errors, traffic will continue to be routed towards the blue environment thus there is near-zero downtime.
If errors are detected after the green environment has been promoted, rolling back to the previous version is straightforward — traffic is redirected back to the blue environment.

Typical blue green deployment. The green environment is provisioned in parallel with the blue environment. When the green environment is ready, the load balancer directs traffic to the green environment.

In addition to blue-green deployments, canary tests and rolling deployments release the new version of the service to a small group of users, thereby reducing the blast radius when the green environment fails.

Canary tests can be done with blue-green environments. After the green environment is ready, the load balancer sends a small fraction of the traffic to the green environment (in this example, 10%).

Canary test/deployment. All traffic is directed to the blue environment initially. When you perform a canary test, 10% of the traffic is directed to the green environment.

If no errors are identified during the canary tests, you incrementally direct traffic to the green environment (50/50 — split traffic). Finally, the load balancer will direct all traffic to the green environment. When you're comfortable, destroy the old, blue environment. The green environment is now the current production service.

![Rolling deployment. After the initial canary test, traffic to the green environment is split evenly with the blue environment (50/50). Finally, all traffic is directed to the green environment.][rolling-deployment]

AWS's application load balancer (ALB) automatically distributes incoming traffic to the appropriate service at the application layer. ALBs are different from classic load balancers which only route traffic to EC2 instances across multiple availability zones. You can define an ALB's listeners (rules) and target groups to dynamically route traffic to services. These rules enable you to run canary tests on and incrementally promote the green environment.

In this tutorial, you will do the following with Terraform:

Provision underlying resources (VPC, security groups, load balancers) and a set of web servers to serve as the blue environment.
Provision a second set of web servers to serve as the green environment.
Add feature toggles to your Terraform configuration to define a list of potential deployment strategies.
Use feature toggles to conduct a canary test and incrementally promote your green environment.

»Prerequisites

This tutorial assumes you are familiar with the standard Terraform workflow. If you are unfamiliar with Terraform, complete the Get Started tutorials first.

For this tutorial, you will need:

Terraform 0.14+ installed locally
an AWS account

»Explore the workspace

Clone the Learn Terraform Advanced Deployment Strategies repository.

$ git clone https://github.com/hashicorp/learn-terraform-advanced-deployments.git

Navigate to the repository directory in your terminal.

$ cd learn-terraform-advanced-deployments

This directory is a Terraform workspace with multiple files:

main.tf contains configuration to deploy this tutorial's VPC, security groups, and load balancers
variables.tf defines variables used by the configuration such as region, CIDR blocks, number of subnets, etc.
blue.tf contains configuration to deploy 2 AWS instances and start a web server. These instances represent a sample application's "version 1.0".
init-script.tf contains the script to start the web server.
versions.tf contains a terraform block which specifies the Terraform binary version and AWS provider version
.terraform.lock.hcl is the Terraform dependency lock file

Note: This workspace combines the network (load balancer) and the application configuration in one directory as a learning exercise. In a production environment, these configuration would be separated, using terraform_remote_state to share data between the workspaces.

»Explore `main.tf`

Open main.tf. This file uses the AWS provider to deploy the base infrastructure for this tutorial, including a VPC, subnets, an application security group, and a load balancer security group.

Toward the end of the file, you'll find aws_lb which represents an ALB. When the load balancer receives the request, it evaluates the listener rules, defined by aws_lb_listener.app, and routes traffic to the appropriate target group.

This load balancer currently directs all traffic to the blue load balancing target group on port 80.

resource "aws_lb" "app" {
  name               = "main-app-lb"
  internal           = false
  load_balancer_type = "application"
  subnets            = module.vpc.public_subnets
  security_groups    = [module.lb_security_group.this_security_group_id]
}

resource "aws_lb_listener" "app" {
  load_balancer_arn = aws_lb.app.arn
  port              = "80"
  protocol          = "HTTP"

  default_action {
    type             = "forward"
    target_group_arn = aws_lb_target_group.blue.arn
  }
}

»Explore `blue.tf`

Open blue.tf. Here you'll find the configuration to deploy 2 AWS instances that start web servers, which return the text Version 1.0 - #${count.index}. This represents the sample application's first version.

resource "aws_instance" "blue" {
  count = var.enable_blue_env ? var.blue_instance_count : 0

  ami                    = data.aws_ami.amazon_linux.id
  instance_type          = "t2.micro"
  subnet_id              = module.vpc.public_subnets[count.index % length(module.vpc.public_subnets)]
  vpc_security_group_ids = [module.app_security_group.this_security_group_id]
  user_data = templatefile("${path.module}/init-script.sh.tftpl", {
    file_content = "version 1.0 - #${count.index}"
  })

  tags = {
    Name = "version-1.0-${count.index}"
  }
}

In addition, this file defines the blue load balancer target group and attaches the blue instances to it using aws_lb_target_group_attachment.

resource "aws_lb_target_group" "blue" {
  name     = "blue-tg-${random_pet.app.id}-lb"
  port     = 80
  protocol = "HTTP"
  vpc_id   = module.vpc.vpc_id

  health_check {
    port     = 80
    protocol = "HTTP"
    timeout  = 5
    interval = 10
  }
}

resource "aws_lb_target_group_attachment" "blue" {
  count            = length(aws_instance.blue)
  target_group_arn = aws_lb_target_group.blue.arn
  target_id        = aws_instance.blue[count.index].id
  port             = 80
}

»Initialize and apply the configuration

In your terminal, initialize your Terraform workspace.

$ terraform init
Initializing modules...
Downloading terraform-aws-modules/security-group/aws 3.17.0 for app_security_group...
- app_security_group in .terraform/modules/app_security_group/modules/web
- app_security_group.sg in .terraform/modules/app_security_group
Downloading terraform-aws-modules/security-group/aws 3.17.0 for lb_security_group...
- lb_security_group in .terraform/modules/lb_security_group/modules/web
- lb_security_group.sg in .terraform/modules/lb_security_group
Downloading terraform-aws-modules/vpc/aws 2.64.0 for vpc...
- vpc in .terraform/modules/vpc

Initializing the backend...

Initializing provider plugins...
- Reusing previous version of hashicorp/aws from the dependency lock file
- Reusing previous version of hashicorp/random from the dependency lock file
- Installing hashicorp/aws v3.20.0...
- Installed hashicorp/aws v3.20.0 (signed by HashiCorp)
- Installing hashicorp/random v3.0.0...
- Installed hashicorp/random v3.0.0 (signed by HashiCorp)

Terraform has been successfully initialized!

## ...

Apply your configuration. Remember to confirm your apply with a yes.

$ terraform apply

## ...

Plan: 36 to add, 0 to change, 0 to destroy.

## ...

Apply complete! Resources: 36 added, 0 changed, 0 destroyed.

Outputs:

lb_dns_name = "main-app-bursting-slug-lb-976734382.us-west-2.elb.amazonaws.com"

»Verify blue environment

Verify that your blue environment was deployed successfully by visiting the load balancer's DNS name in your browser or cURLing it from your terminal.

Note: It may take a few minutes for the load balancer's health checks to be successful and the web servers to respond.

$ for i in `seq 1 5`; do curl $(terraform output -raw lb_dns_name); done
Version 1.0 - #0!
Version 1.0 - #1!
Version 1.0 - #0!
Version 1.0 - #0!
Version 1.0 - #1!

Notice that the load balancer is routing traffic evenly to both VMs in the blue environment.

»Deploy green environment

Create a new file named green.tf and paste in the configuration for the sample application's version 1.1.

resource "aws_instance" "green" {
  count = var.enable_green_env ? var.green_instance_count : 0

  ami                    = data.aws_ami.amazon_linux.id
  instance_type          = "t2.micro"
  subnet_id              = module.vpc.public_subnets[count.index % length(module.vpc.public_subnets)]
  vpc_security_group_ids = [module.app_security_group.this_security_group_id]
  user_data = templatefile("${path.module}/init-script.sh.tftpl", {
    file_content = "version 1.1 - #${count.index}"
  })

  tags = {
    Name = "green-${count.index}"
  }
}

resource "aws_lb_target_group" "green" {
  name     = "green-tg-${random_pet.app.id}-lb"
  port     = 80
  protocol = "HTTP"
  vpc_id   = module.vpc.vpc_id

  health_check {
    port     = 80
    protocol = "HTTP"
    timeout  = 5
    interval = 10
  }
}

resource "aws_lb_target_group_attachment" "green" {
  count            = length(aws_instance.green)
  target_group_arn = aws_lb_target_group.green.arn
  target_id        = aws_instance.green[count.index].id
  port             = 80
}

Notice how this configuration is similar to the blue application, except that the web servers return green #${count.index}.

Add the following variables to variables.tf.

variable "enable_green_env" {
  description = "Enable green environment"
  type        = bool
  default     = true
}

variable "green_instance_count" {
  description = "Number of instances in green environment"
  type        = number
  default     = 2
}

Apply your configuration to deploy your green application. Remember to confirm your apply with a yes.

$ terraform apply

## ...

Plan: 5 to add, 0 to change, 0 to destroy.

## ...

Apply complete! Resources: 5 added, 0 changed, 0 destroyed.

Outputs:

lb_dns_name = "main-app-bursting-slug-lb-976734382.us-west-2.elb.amazonaws.com"

»Add feature toggles to route traffic

Even though you deployed your green environment, the load balancer is currently not routing traffic to it.

$ for i in `seq 1 5`; do curl $(terraform output -raw lb_dns_name); done
Version 1.0 - #1!
Version 1.0 - #0!
Version 1.0 - #0!
Version 1.0 - #0!
Version 1.0 - #1!

While you could manually modify the load balancer's target groups to include the green environment, using feature toggles codifies this change for you. In this step, you will add a traffic_distribution variable and traffic_dist_map local variable to your configuration. The configuration will automatically assign the target group's weight based on the traffic_distribution variable.

First, add the configuration for the local value and traffic distribution variable to variables.tf.

locals {
  traffic_dist_map = {
    blue = {
      blue  = 100
      green = 0
    }
    blue-90 = {
      blue  = 90
      green = 10
    }
    split = {
      blue  = 50
      green = 50
    }
    green-90 = {
      blue  = 10
      green = 90
    }
    green = {
      blue  = 0
      green = 100
    }
  }
}

variable "traffic_distribution" {
  description = "Levels of traffic distribution"
  type        = string
}

Notice that there are five traffic distributions defined by the local variable. Each traffic distribution specifies the weight for the respective target group.

The blue target distribution is what is currently being applied — 100% of the traffic is routed to the blue environment, 0% to the green environment. The blue-90 target distribution simulates canary testing. This canary test will route 90% of the traffic to the blue environment and 10% to the green environment. The split target distribution builds on top of canary testing by increasing traffic to the green environment. This will split the traffic evenly between the blue and green environments (50/50). The green-90 target distribution increases traffic to the green environment — 90% of the traffic is routed to the green environment, 10% to the blue environment. The green target distribution will completely promote the green environment — the load balancer will route 100% of the traffic to the green environment.

Replace the aws_lb_listener.app default action block in main.tf. The configuration uses lookup to set the target groups' weight. Notice that the configuration defaults to a directing all traffic to the blue environment if no value is set.

Modify the aws_lb_listener.app's default action block in main.tf to match the following. The configuration uses lookup to set the target groups' weight. Notice that the configuration defaults to a directing all traffic to the blue environment if no value is set.

resource "aws_lb_listener" "app" {
  ## ...
    default_action {
      type             = "forward"
-      target_group_arn = aws_lb_target_group.blue.arn
+      forward {
+        target_group {
+          arn    = aws_lb_target_group.blue.arn
+          weight = lookup(local.traffic_dist_map[var.traffic_distribution], "blue", 100)
+        }

+        target_group {
+          arn    = aws_lb_target_group.green.arn
+          weight = lookup(local.traffic_dist_map[var.traffic_distribution], "green", 0)
+        }

+        stickiness {
+          enabled  = false
+          duration = 1
+        }
+      }
    }
}

Note: The ELB's stickiness is set to false and 1 second to demonstrate how traffic is directed between the two target groups. Refer to this AWS blog for guidance on connection draining and stickiness settings for production environments.

»Start shifting traffic to green environment

Apply your configuration to run a canary test by setting the traffic_distribution variable to blue-90. Remember to confirm your apply with a yes.

$ terraform apply -var 'traffic_distribution=blue-90'

## ...

Terraform will perform the following actions:

  # aws_lb_listener.app will be updated in-place
  ~ resource "aws_lb_listener" "app" {
        id                = "arn:aws:elasticloadbalancing:us-west-2:561656980159:listener/app/main-app-bursting-slug-lb/ace5fc1e7af739e9/1823f9a776b1232e"
        # (4 unchanged attributes hidden)

      ~ default_action {
          - target_group_arn = "arn:aws:elasticloadbalancing:us-west-2:561656980159:targetgroup/blue-tg-bursting-slug-lb/c8ed1be403ce253c" -> null
            # (2 unchanged attributes hidden)

          + forward {
              + stickiness {
                  + duration = 1
                  + enabled  = false
                }

              + target_group {
                  + arn    = "arn:aws:elasticloadbalancing:us-west-2:561656980159:targetgroup/blue-tg-bursting-slug-lb/c8ed1be403ce253c"
                  + weight = 100
                }
              + target_group {
                  + arn    = "arn:aws:elasticloadbalancing:us-west-2:561656980159:targetgroup/green-tg-bursting-slug-lb/ade9778242aab1c2"
                  + weight = 0
                }
            }
        }
    }

Plan: 0 to add, 1 to change, 0 to destroy.

## ...

aws_lb_listener.app: Modifying... [id=arn:aws:elasticloadbalancing:us-west-2:561656980159:listener/app/main-app-bursting-slug-lb/ace5fc1e7af739e9/1823f9a776b1232e]
aws_lb_listener.app: Modifications complete after 0s [id=arn:aws:elasticloadbalancing:us-west-2:561656980159:listener/app/main-app-bursting-slug-lb/ace5fc1e7af739e9/1823f9a776b1232e]

Apply complete! Resources: 0 added, 1 changed, 0 destroyed.

Outputs:

lb_dns_name = "main-app-bursting-slug-lb-976734382.us-west-2.elb.amazonaws.com"

»Verify canary deployment traffic

Verify that your load balancer is now routing 10% of the traffic to the green environment.

Note: It may take a few minutes for the load balancer's health checks to be successful and for the green environment to begin responding.

$ for i in `seq 1 10`; do curl $(terraform output -raw lb_dns_name); done
Version 1.0 - #0!
Version 1.0 - #0!
Version 1.0 - #1!
Version 1.0 - #0!
Version 1.1 - #1!
Version 1.0 - #0!
Version 1.0 - #0!
Version 1.0 - #1!
Version 1.0 - #0!
Version 1.0 - #1!

Notice that 10% of the traffic was routed to the green environment.

»Increase traffic to green environment

Now that the canary deployment was successful, increase the traffic to the green environment.

Apply your configuration to run a canary test by setting the traffic_distribution variable to split. Remember to confirm your apply with a yes.

$ terraform apply -var 'traffic_distribution=split'

## ...

Terraform will perform the following actions:

  # aws_lb_listener.app will be updated in-place
  ~ resource "aws_lb_listener" "app" {
        id                = "arn:aws:elasticloadbalancing:us-west-2:561656980159:listener/app/main-app-bursting-slug-lb/ace5fc1e7af739e9/1823f9a776b1232e"
        # (4 unchanged attributes hidden)

      ~ default_action {
            # (2 unchanged attributes hidden)

          ~ forward {

              + target_group {
                  + arn    = "arn:aws:elasticloadbalancing:us-west-2:561656980159:targetgroup/blue-tg-bursting-slug-lb/c8ed1be403ce253c"
                  + weight = 50
                }
              - target_group {
                  - arn    = "arn:aws:elasticloadbalancing:us-west-2:561656980159:targetgroup/blue-tg-bursting-slug-lb/c8ed1be403ce253c" -> null
                  - weight = 90 -> null
                }
              - target_group {
                  - arn    = "arn:aws:elasticloadbalancing:us-west-2:561656980159:targetgroup/green-tg-bursting-slug-lb/ade9778242aab1c2" -> null
                  - weight = 10 -> null
                }
              + target_group {
                  + arn    = "arn:aws:elasticloadbalancing:us-west-2:561656980159:targetgroup/green-tg-bursting-slug-lb/ade9778242aab1c2"
                  + weight = 50
                }
                # (1 unchanged block hidden)
            }
        }
    }

Plan: 0 to add, 1 to change, 0 to destroy.

## ...

Apply complete! Resources: 0 added, 1 changed, 0 destroyed.

Outputs:

lb_dns_name = "main-app-bursting-slug-lb-976734382.us-west-2.elb.amazonaws.com"

»Verify rolling deployment traffic

Verify that your load balancer is now splitting the traffic to the blue and green environments.

Note: It may take a few minutes for the load balancer's health checks to be successful. The results may not always show a clean 50/50 split when running the above command.

$ for i in `seq 1 10`; do curl $(terraform output -raw lb_dns_name); done
Version 1.0 - #0!
Version 1.1 - #1!
Version 1.0 - #1!
Version 1.1 - #1!
Version 1.1 - #0!
Version 1.0 - #0!
Version 1.0 - #1!
Version 1.0 - #1!
Version 1.1 - #1!
Version 1.1 - #0!

Notice how 50% of the traffic was routed to the blue environment and the rest to the green environment.

»Promote green environment

Now that both the canary and rolling deployments were successful, route 100% of the load balancer's traffic to the green environment to promote it.

Apply your configuration to promote the green environment by setting the traffic_distribution variable to green. Remember to confirm your apply with a yes.

$ terraform apply -var 'traffic_distribution=green'

## ...

Terraform will perform the following actions:

  # aws_lb_listener.app will be updated in-place
  ~ resource "aws_lb_listener" "app" {
        id                = "arn:aws:elasticloadbalancing:us-west-2:561656980159:listener/app/main-app-bursting-slug-lb/ace5fc1e7af739e9/1823f9a776b1232e"
        # (4 unchanged attributes hidden)

      ~ default_action {
            # (2 unchanged attributes hidden)

          ~ forward {

              + target_group {
                  + arn    = "arn:aws:elasticloadbalancing:us-west-2:561656980159:targetgroup/blue-tg-bursting-slug-lb/c8ed1be403ce253c"
                  + weight = 0
                }
              - target_group {
                  - arn    = "arn:aws:elasticloadbalancing:us-west-2:561656980159:targetgroup/blue-tg-bursting-slug-lb/c8ed1be403ce253c" -> null
                  - weight = 50 -> null
                }
              + target_group {
                  + arn    = "arn:aws:elasticloadbalancing:us-west-2:561656980159:targetgroup/green-tg-bursting-slug-lb/ade9778242aab1c2"
                  + weight = 100
                }
              - target_group {
                  - arn    = "arn:aws:elasticloadbalancing:us-west-2:561656980159:targetgroup/green-tg-bursting-slug-lb/ade9778242aab1c2" -> null
                  - weight = 50 -> null
                }
                # (1 unchanged block hidden)
            }
        }
    }

Plan: 0 to add, 1 to change, 0 to destroy.

## ...

Apply complete! Resources: 0 added, 1 changed, 0 destroyed.

Outputs:

lb_dns_name = "main-app-bursting-slug-lb-976734382.us-west-2.elb.amazonaws.com"

»Verify load balancer traffic

Verify that your load balancer is routing all traffic to the green environment.

Note: It may take a few minutes for the load balancer's health checks to be successful.

$ for i in `seq 1 5`; do curl $(terraform output -raw lb_dns_name); done
Version 1.1 - #1!
Version 1.1 - #0!
Version 1.1 - #0!
Version 1.1 - #0!
Version 1.1 - #1!

Congrats! You have successfully promoted your green environment with near-zero downtime.

»Scale down blue environment

Now that you have verified that all traffic is directed to your green environment, it is safe to disable the blue environment.

Apply your configuration to disable the blue environment by setting the traffic_distribution variable to green and enable_blue_env to false. Remember to confirm your apply with a yes.

$ terraform apply -var 'traffic_distribution=green' -var 'enable_blue_env=false'

»Deploy new version

In this tutorial, you deployed the application's Version 1.0 on the blue environment, and the new version, 1.1, on the green environment. When you promoted the green environment, it became the current production environment. Deploy the next release on the blue environment, which minimizes modifications to your existing configuration by alternating the blue and green environments.

Alternating blue-green deployments. Version 1.0 - Blue; Version 1.1 - Green; Version 1.2 - Blue; Version 1.3 - Green...

When you're ready to deploy a new application version, update the blue environment instances.

Modify the aws_instance.blue's user_data and tags blocks in blue.tf to display a new version number, 1.2.

resource "aws_instance" "blue" {
  ## ...

  user_data = templatefile("${path.module}/init-script.sh.tftpl", {
-    file_content = "version 1.0 - #${count.index}"
+    file_content = "version 1.2 - #${count.index}"
  })

  tags = {
-    Name = "version-1.0-${count.index}"
+    Name = "version-1.2-${count.index}"
  }
}

»Enable new version environment

Apply your configuration to provision the new version of your infrastructure. Remember to confirm your apply with a yes. Notice that the traffic_distribution variable is set to green. This is because the green environment contains your current production environment.

$ terraform apply -var 'traffic_distribution=green'

»Start shifting traffic to blue environment

Now that the new version is deployed to the blue environment, start shifting traffic to it.

Apply your configuration to run a canary test by setting the traffic_distribution variable to green-90. Remember to confirm your apply with a yes.

$ terraform apply -var 'traffic_distribution=green-90'

Once the apply completes, Verify that your load balancer is routing all traffic to the green environment.

Note: It may take a few minutes for the load balancer's health checks to be successful.

$ for i in `seq 1 10`; do curl $(terraform output -raw lb_dns_name); done
Version 1.1 - #1!
Version 1.1 - #0!
Version 1.1 - #0!
Version 1.2 - #0!
Version 1.1 - #1!
Version 1.1 - #1!
Version 1.1 - #0!
Version 1.1 - #0!
Version 1.1 - #1!
Version 1.1 - #0!

»Promote blue environment

Now that the canary deployment is successful, fully promote your blue environment.

Apply your configuration to promote the blue environment by setting the traffic_distribution variable to blue. Remember to confirm your apply with a yes.

$ terraform apply -var 'traffic_distribution=blue'

Verify that your load balancer is routing all traffic to the blue environment.

Note: It may take a few minutes for the load balancer's health checks to be successful.

$ for i in `seq 1 5`; do curl $(terraform output -raw lb_dns_name); done
Version 1.2 - #1!
Version 1.2 - #0!
Version 1.2 - #0!
Version 1.2 - #0!
Version 1.2 - #1!

Congrats! You have successfully used blue-green, canary and rolling deployments to schedule two releases.

»Clean up your infrastructure

After verifying that the resources were deployed successfully, destroy them. Remember to respond to the confirmation prompt with yes.

$ terraform destroy -var 'traffic_distribution=blue'

»Next steps

In this tutorial, you used an ALB to incrementally deploy a new application release. In addition, you implemented feature toggles to codify and run these advanced deployments techniques in a consistent, reliable manner.

To automate your infrastructure deployment process and to learn more about using load balancers to schedule near-zero downtime releases, visit the following resources.

The Deploy Terraform infrastructure with CircleCI tutorial
The Blue/Green & Canary Deployments Nomad tutorial
The Traffic Splitting for Service Deployments Consul tutorial
The Fine-tuning blue/green deployments on application load balancer AWS blog post walks you through blue green deployments using application load balancers. This blog post dives into more details on how to set target group level stickiness and enable connection draining to provide near-zero downtime releases.

Infrastructure

Security

Networking

Applications

»Prerequisites

»Explore the workspace

»Explore main.tf

»Explore blue.tf

»Initialize and apply the configuration

»Verify blue environment

»Deploy green environment

»Add feature toggles to route traffic

»Start shifting traffic to green environment

»Verify canary deployment traffic

»Increase traffic to green environment

»Verify rolling deployment traffic

»Promote green environment

»Verify load balancer traffic

»Scale down blue environment

»Deploy new version

»Enable new version environment

»Start shifting traffic to blue environment

»Promote blue environment

»Clean up your infrastructure

»Next steps

»Explore `main.tf`

»Explore `blue.tf`