HashiCorp Cloud Platform (HCP) is a fully managed platform offering HashiCorp Products as a Service (HPaaS) to automate infrastructure on any cloud.
This tutorial will cover the process required to connect an Amazon ECS cluster using EC2 instances to HCP Consul on AWS.
»Prerequisites
- An HCP Consul deployment with a public endpoint (for debugging purposes only)
- Consul 1.9+
- AWS CLI
- jq
- An Amazon VPC with public and private subnets
- An Amazon ECS Cluster using EC2 Instances
- A set of three Amazon ECS Container instances with
awsvpcmode and AWS Secrets Manager enabled. Ensure the following cluster attributes are set:
For this tutorial, you will need to ensure your AWS CLI is logged in with your credentials, and is targeting the region where you have created your ECS cluster. Review the AWS documentation for instructions on how to configure the AWS CLI.
To enable communication between the Consul control plane and the ECS cluster, you will need to complete the process outlined in the manual deployment tutorial or in the Terraform deployment tutorial.
NOTE: This tutorial uses the joatmon08/consul-ecs image for the Consul clients and proxy sidecars. This image is not officially supported by HashiCorp. It abstracts some of the configuration for this tutorial and it is not recommended for production use. Examine the source code on GitHub to customize additional parameters.
»Allow Amazon ECS to communicate with HCP Consul
The container instances that make up the Amazon ECS cluster need to be able to communicate with HCP Consul over the required Consul ports.
Set the name of your ECS cluster as the ECS_CLUSTER_NAME environment variable
so you can reference it in commands throughout the tutorial.
Retrieve the ID of the security group associated with your ECS container instances,
and set it to the ECS_SECURITY_GROUP_ID environment variable.
Verify that the security group for the ECS container instances include the following rules that allow LAN, Serf, RPC, HTTPS, and GRPC traffic.
»Configure development host
Now, download the Consul client configuration from the HCP portal by clicking the "Download client config" button from within the portal.
Unzip the client configuration file to a directory named client_config.
Change directories to client_config.
Confirm that there are both client_config.json and ca.pem files.
From this same screen in the HCP UI, click the "Generate token" button and then click "Copy" from the dialog box. A global-management root token is now in your clipboard.
Set it to the CONSUL_HTTP_TOKEN environment variable on your
development host so that you can reference it later in the tutorial.
»Configure Consul secrets
Consul Service on HCP is secure by default. This means that client agents will need to be configured with the gossip encryption key, the Consul CA cert, and a bootstrap ACL token. All three of these secrets will need to be stored in the AWS Secrets Manager so that they can be referenced and retrieved during ECS service deployment.
From the unzipped client config directory, base64 encode the ca.pem file
and store it into an environment variable named CONSUL_CA_PEM. Encoding the Consul
CA cert allows AWS Secrets Manager to inject the secret into the ECS task.
Extract the gossip encryption key from the client configuration.
Extract the private Consul address from client configuration.
Extract the Consul datacenter from the client configuration.
You will write a here document for JSON files throughout this tutorial. The here document will interpolate the environment variables you set in your shell as part of this tutorial into the defined string values.
Write a here document that evaluates the CONSUL_HTTP_ADDR, CONSUL_HTTP_TOKEN,
CONSUL_CA_PEM, and CONSUL_GOSSIP_ENCRYPT environment variables to a file
named consul-client-secret.json for the Consul certificate
authority, gossip encrypt key, server endpoint, and token.
Verify that that the JSON configuration in consul-client-secret.json contains
the Consul token, CA, HTTP address, and gossip encryption key in plaintext.
Using the AWS CLI, create a secret named consul and set the value
to read from consul-client-secret.json. Save the ARN of the secret into
CONSUL_CLIENT_SECRET_ARN.
»Create the AWS IAM Role for Consul client ECS tasks
In order for Amazon ECS tasks to access AWS Secrets Manager, you need to create and assign AWS IAM roles that allow the task to retrieve a specific secret. This is done with a task role. The client must use a task role to ensure they cannot access any secrets other than the client secret.
For configuration such as AWS IAM roles or ECS task definitions in this tutorial, write a here document to a file. The here document will interpolate the environment variables you set in your shell as part of this tutorial into the defined string values. You will set environment variables that capture identifiers for the ECS cluster, security group, networking, task definitions, and load balancers. After writing the here document to a file, you can reference the file with the AWS CLI.
Write a here document for an AWS IAM policy to a file named ecs-trusted-entity.json.
The policy allows the ECS task to assume a role.
Write a here document which interpolates the CONSUL_CLIENT_SECRET_ARN environment
variable into a file named consul-client-policy.json. This policy allows an AWS IAM
role to access the secrets for the Consul clients in AWS Secrets Manager.
Create an AWS IAM role that uses the trusted entity policy in ecs-trusted-entity.json.
Attach the consul-client-policy.json as an embedded policy to the role.
»Create the ECS task definition for Consul clients
After setting up the secrets and enabling ECS tasks access to read them, you can
create the task definitions for Consul clients. You must add the role and secret
ARNs to the task definitions for the Consul clients to run properly.
Use the consul-ecs container to start the client
with a few required environment variables. The client runs in host networking mode so that
it can be deployed as one per ECS container instance. The Consul ACL token for registering client
will be set as the CONSUL_HTTP_TOKEN environment variable within the sidecar.
Write a here document to a file named consul-client-definition.json. The here document
outlines an ECS task definition and interpolates the CONSUL_CLIENT_ROLE_ARN,
CONSUL_DATACENTER, and CONSUL_CLIENT_SECRET_ARN environment variables into the JSON.
Register the task definition in consul-client-definition.json.
»Deploy the Consul clients
Now, deploy the Consul clients to ECS with the DAEMON scheduling strategy.
This will deploy one Consul client per ECS container host in the cluster.
Verify that the Consul clients have deployed successfully by checking that the desired task count matches the running count.
If you deployed HCP Consul with a public endpoint, copy the public URL and set to the
CONSUL_HTTP_ADDR environment variable.
Verify that the Consul clients in ECS are registered with HCP Consul.
»Configure secrets for an example workload
Now that you have deployed the Consul clients, deploy an application workload. This tutorial uses the Counting demo application.
Create a Consul ACL token that will represent the counting service's identity
in Consul and use jq to extract the token's value stored in SecretID.
Create a Consul ACL token that will represent the dashboard service's identity
in Consul and use jq to extract the token's value stored in SecretID.
Amazon ECS needs access to the ACL tokens as part of the task definition. As a result, you must store the service's ACL tokens into a secrets manager, in this case AWS Secrets Manager.
Using the AWS CLI, create a secret named counting and set the value
to the COUNTING_ACL_TOKEN environment variable.
Using the AWS CLI, create a secret named dashboard and set the value
to the DASHBOARD_ACL_TOKEN environment variable.
»Configure AWS IAM roles for an example workload
In order for Amazon ECS tasks to access AWS Secrets Manager, you must create AWS IAM roles that allow the task to retrieve a specific secret. This is done with a task role. Each service must be configured with a different task role to ensure they can only access their own Consul ACL token.
Write a here document for an AWS IAM policy to a file named ecs-trusted-entity.json.
The policy allows the ECS task to assume a role.
Write a here document which evaluates the COUNTING_SECRET_ARN environment
variable and saves it in a file named counting-policy.json. This policy
allows an AWS IAM role to access the secrets for the counting service in
AWS Secrets Manager.
Write a here document which evaluates the DASHBOARD_SECRET_ARN environment
variable and saves it in a file named dashboard-policy.json. This policy
allows an AWS IAM role to access the secrets for the dashboard service in
AWS Secrets Manager.
Create an AWS IAM role for counting that uses the trusted entity policy
in ecs-trusted-entity.json
Attach the counting-policy.json as an embedded policy.
Create an AWS IAM role for dashboard that uses the trusted entity policy
in ecs-trusted-entity.json
Attach the dashboard-policy.json as an embedded policy.
»Create the ECS task definitions for an example workload
After setting up the secrets and enabling the ECS task's access to read them, you can create the task definitions for the counting and dashboard services. You must add the role and secret ARNs to the task definitions for the counting and dashboard services in order to run them.
The counting service needs two containers, one for the counting service on port 9001 and
another for the counting service's sidecar listening on port 21000. The Consul ACL token
for registering the counting service will be set as the CONSUL_HTTP_TOKEN environment
variable within the sidecar.
Write a here document to a file named counting-definition.json. The here document
outlines an ECS task definition and interpolates the COUNTING_ROLE_ARN and
COUNTING_SECRET_ARN environment variables into the JSON.
Register the task definition in counting-definition.json.
The dashboard service needs two containers, one for the dashboard service on port 9002
and another for the dashboard service's sidecar listening on port 21000. The Consul
ACL token for registering the dashboard service will be set as the CONSUL_HTTP_TOKEN
environment variable within the sidecar.
The main difference between the dashboard and counting services is that dashboard accesses
the upstream counting service endpoint. Configure this by specifying the upstream
using the CONSUL_SERVICE_UPSTREAMS environment variable. The upstream configuration must be
a JSON blob with no spaces and escaped double quotes.
Write a here document to a file named dashboard-definition.json. The here document
outlines an ECS task definition and interpolates the DASHBOARD_ROLE_ARN and
DASHBOARD_SECRET_ARN environment variables into the JSON.
Register the task definition in dashboard-definition.json.
»Deploy an example workload
Before deploying the counting service, retrieve the VPC ID, subnet IDs,
and security group IDs for your EC2 container instances. You need this
information because the counting service runs in awsvpc mode,
which means you must specify the networking configuration.
Retrieve all of the subnet ids associated with ECS container instances.
Retrieve the VPC ID for the cluster subnets.
Create one instance of the counting service, and add the subnets and security groups related to ECS.
Verify that the counting service is registered with Consul using the Consul CLI.
Create a security group for a load balancer. It will forward requests to the dashboard service.
Authorize a security group rule from your development environment to the load balancer.
Remove the outbound security group since it is too permissive.
Replace the egress rule to only allow traffic to the ECS cluster.
Get a list of public subnets for the ECS cluster's VPC.
Create the application load balancer to forward requests to the dashboard service.
Create a target group for the dashboard service, which will route traffic to dashboard.
The target-type must be ip due to awsvpc mode.
Create a listener that forwards traffic to the target group. Make sure you have defined
the ${ALB_ARN} with the ARN of the application load balancer.
Create one instance of the dashboard service and add the subnets and security groups related to ECS.
Verify that the dashboard service registered with Consul using the Consul CLI.
»Create the necessary intentions
Since HCP Consul on AWS is secure by default, the datacenter is created with a "default deny" intention in place. This means that, by default, no services can interact with each other until an operator explicitly allows them to do so by creating intentions for each inter-service operations they wish to allow.
Issue the following command to create an intention to allow dashboard to connect to counting.
»Access the dashboard service
Get the endpoint for your load balancer.
Verify you have successfully deployed the application by visiting the load balancer's DNS name in a browser tab. This validates that Consul service discovery is working, because the dashboard service resolves to the counting service. This also validates that Consul service mesh is working, because the intentions that were created are allowing services to interact with one another.
»Next Steps
In this tutorial, you deployed a set of Consul clients to an Amazon ECS cluster using EC2 instances and deployed a demo application. To learn more, complete the following tutorials.
If you encounter any issues, please contact the HCP team at support.hashicorp.com.
