Consul service mesh deploys an Envoy sidecar proxy alongside each service instance in a datacenter. The sidecar proxy brokers traffic between the local service instance and other services registered with Consul. The proxy is aware of all traffic that passes through it. In addition to securing inter-service communication, the proxy can also collect and expose data about the service instance. Starting with version 1.5, Consul service mesh is able to configure Envoy to expose layer 7 metrics, such as HTTP status codes or request latency, to monitoring tools like Prometheus.
In this tutorial, you will:
- Configure Consul to expose Envoy metrics to Prometheus
- Deploy Consul using the official helm chart.
- Deploy Prometheus and Grafana using their official Helm charts.
- Deploy a multi-tier demo application that is configured to be scraped by Prometheus.
- Start a traffic simulation deployment, and observe the application traffic in Grafana.
Tip: While this tutorial shows you how to deploy a metrics pipeline on Kubernetes, all the technologies the tutorial uses are platform agnostic; Kubernetes is not necessary to collect and visualize layer 7 metrics with Consul service mesh.
»Prerequisites
If you already have a Kubernetes cluster running with Helm and kubectl installed, you can start on the tutorial right away. If not, set up a Kubernetes cluster using your favorite method that supports persistent volume claims, or install and start Minikube v1.10.1+ or kind v0.8.1+.
If you use Minikube, you may want to start it with a little bit of extra memory.
You must also install kubectl, and both install and initialize Helm.
Also, ensure you have the latest helm charts for Consul, Prometheus, and Grafana.
Next, clone the GitHub repository that contains the files you'll use with this tutorial.
Now, change directories into the repository you just cloned.
Now, checkout the tagged version verified for this tutorial.
We'll refer to this directory as your working directory, and you'll run the rest of the commands in this tutorial from this directory.
»Deploy Consul service mesh using Helm
Once you have set up the prerequisites, you're ready to install Consul.
Open the file in your working directory called layer7-observability/helm/consul-values.yaml and
review the settings. You can override many of the values in Consul's values file
using annotations on specific services.
Note the extraConfig and proxyDefaults entries. The extraConfig entry injects the required Consul settings to enable the Prometheus metrics provider. The proxyDefaults entry injects a proxy-defaults Consul configuration entry
for the envoy_prometheus_bind_addr setting that is applied to all Envoy proxy instances.
Consul then uses that setting to configure where Envoy will publish Prometheus metrics.
This is important because you will need to annotate your pods with this port so that Prometheus
can scrape them. We will cover this in more detail later in the tutorial.
Warning: By default, the chart will install an insecure configuration of Consul. This provides a less complicated out-of-box experience for new users but is not appropriate for a production setup. Review the Secure Consul and Registered Services on Kubernetes tutorial for instructions on how to secure your datacenter for production.
Now install Consul in your Kubernetes cluster and give Kubernetes a name for your Consul installation.
The output will be a list of all the Kubernetes resources created. The output has been abbreviated, but will look similar to the following.
Check that Consul is running in your Kubernetes cluster using kubectl.
Consul setup is complete when all pods have a status of Running, as
illustrated in the following output.
Leave the watch running and open a new terminal.
»Deploy the metrics pipeline
Consul service mesh can integrate with a variety of metrics tooling, but in this tutorial, you will use Prometheus and Grafana to collect and visualize metrics.
»Deploy Prometheus with Helm
Install the official Prometheus Helm chart using the values in
layer7-observability/helm/prometheus-values.yaml.
The output has been abbreviated, but will look similar to the following.
Switch back to the terminal where the kubectl watch is running. Prometheus setup
is complete when all pods have a status of Running, as illustrated in the following
output.
»Deploy Grafana with Helm
Installing Grafana will follow a similar process. Install the
official Grafana Helm chart using the values in
layer7-observability/helm/grafana-values.yaml. This configuration will tell Grafana
to use Prometheus as a datasource, and set the admin password to
password.
The output has been abbreviated, but will look similar to the following.
Switch back to the terminal where the kubectl watch is running. Grafana setup
is complete when the Grafana pod has a status of Running, as illustrated in the following
output.
The output includes shell-specific instructions to access your Grafana UI and
admin password. You specified the admin password in the layer7-observability/helm/grafana-values.yaml
file as password. To expose the Grafana UI outside the cluster, issue
the following command.
Example output:
Leave this port-forward session active so that you can visit the UI again later once metrics are being collected.
Navigate to http://localhost:3000 in a browser tab and log in to the Grafana UI.
Once you have logged into the Grafana UI, hover over the dashboards icon (four
squares in the left hand menu) and then click the "Manage" option.
This will take you to a page that gives you some choices about how to upload
Grafana dashboards. Click the "Import" button on the right hand side of the
screen. Open the file called layer7-observability/grafana/hashicups-dashboard.json
and copy the contents into the JSON window of the Grafana UI. Click through the
rest of the options, and you will end up with a dashboard waiting for data to display.
»Deploy a demo application on Kubernetes
Now that your monitoring pipeline is set up, deploy a demo application that will generate data. We will be using HashiCups, an application that emulates an online order app for a coffee shop. For this tutorial, the HashiCups application includes a React front end, a GraphQL API, a REST API and a Postgres database.
All the files defining HashiCups are in the layer7-observability/hashicups directory.
Open the layer7-observability/hashicups/frontend.yaml
file. Notice that the Deployment has been annotated with
the following Prometheus configuration.
All of the other resources have been similarly annotated. This allows
Prometheus to discover resources in the Kubernetes cluster that should
be exposing metrics, and tells Prometheus what port to the metrics are
exposed at. Recall that earlier in the tutorial you configured the
proxyDefaults entry in the consul-values.yaml file, and set the
envoy_prometheus_bind_addr to 0.0.0.0:9102. By setting this annotation
on each resource, you are configuring the other side of the feedback
loop. It may be helpful to think of this as a publish/subscribe
pattern, where you told Consul to configure Envoy to publish metrics
on port 9102, and now have told Prometheus to subscribe on each proxy
at port 9102. You will verify this configuration is applied later in
the tutorial.
Open a new terminal, and deploy the demo application.
HashiCups will take a few minutes to deploy. Switch back to the terminal
where the kubectl watch is running. Grafana setup is complete when all
pods have a status of Running, as illustrated in the following output.
Test the application by viewing the React front end. You can do this by forwarding the frontend deployment's port 80 to your development host.
Navigate to http://localhost:8080 in a browser window. You should observe the following screen.
Now that the application is running let's verify that Consul did, in fact,
configure Envoy to publish metrics at port 9102. The Envoy side-car proxy
can be reached at port 19000. Open a new terminal, and issue the following
command.
Navigate to http://localhost:19000/config_dump in a browser window. You should
see what looks like a raw JSON document dumped to the screen. This is the Envoy
configuration. Search for 9102 and you should find two different stanzas that
reference this port. One of them is included next for reference.
Notice that the configuration matches the values you specified in the
proxyDefaults stanza in the layer7-observability/helm/consul-values.yaml file.
This confirms that Consul has configured Envoy to publish Prometheus metrics.
Enter CTRL-C to stop the port-forward session from the side-car proxy. You
will not need to reference it again for the remainder of the tutorial.
»Visualize application metrics
While Grafana is optimized for at a glance observability, the Prometheus UI can be useful as well. Issue the following command to expose the Prometheus UI to your development host.
»Discover available metrics with Prometheus
Navigate to http://localhost:9090 in a browser window, and you should observe the default
Prometheus UI. In the textbox at the top of the screen paste sum by(__name__)({app="product-api"})
and then click the button labeled "Execute". Your screen will now look similar
to the following.
You have now performed a PromQL
query that will list all available metrics for resources that have the product-api label.
In this case, that is the REST API resource you deployed from the app folder. This list of metrics
can be used as constraints for further PromQL queries both here and in the Grafana UI.
Next, from within the Prometheus UI, click the dropdown menu item labeled "Status" in the menu bar at the top of the screen, and click on the option labeled "Targets". This will navigate the UI to the Targets screen where you can review all of the resources that Prometheus is monitoring. The Prometheus Helm chart you installed earlier is configured to monitor all of the Kubernetes infrastructure as well as Prometheus itself.
Click on the button labeled "show less" next to each section header except the one labeled "kubernetes-pods". Your screen should appear similar to the following.
Notice that four pods are running, and have a State of "UP". Look at the first label
for each pod in the Label column. You should have entries for app="frontend", app="postgres",
app="products-api", and app="public-api". This confirms that Prometheus is collecting
metrics from all of your annotated resources, and that they are all up and running.
Go back to the terminal where your port-forward session for the Prometheus UI is running,
and type CTRL-C to end the session. You will not need to access the Prometheus UI for
the remainder of the tutorial.
»Simulate traffic
Now that you know the application is running, start generating some load so that you will have some metrics to look at in Grafana.
Envoy exposes a huge number of metrics. Which metrics are important to monitor will depend on your application. For this tutorial we have preconfigured a HashiCups-specific Grafana dashboard with a couple of basic metrics, but you should systematically consider what others you will need to collect as you move from testing into production.
»View Grafana dashboard
Now that you have metrics flowing through your pipeline, and a traffic simulation deployment running, navigate back to your Grafana dashboard in the Grafana UI, and you should observe a screen similar to the following.
Notice that once you started the traffic simulation deployment Prometheus started to log active connections. This dashboard is simplistic, but illustrates that metrics are flowing through the pipeline, and should give you a reasonable starting point for setting up your own observability tooling.
»Viewing traffic in Consul UI
You can also view service metrics in the Consul UI. In a new terminal session, issue the following command to expose the Consul UI to the development host.
Open localhost:8500 in a new browser tab, and navigate
to the "Services" screen. Select the postgres service. You should observe that
a chart with some basic metrics is embedded in the service tile.
Hover over the four tile elements to get tooltip descriptions of the different metrics.
You can also hover the timeline chart embedded in the tile, and review additional metrics for any point in time during the last 15 minutes.
»Clean up
If you've been using Minikube, you can tear down your environment by issuing the following command.
If you want to get rid of the configuration files and Consul Helm chart, recursively remove the learn-consul-kubernetes directory.
`
»Next steps
In this tutorial, you set up layer 7 metrics collection and visualization in a Kubernetes cluster using Consul service mesh, Prometheus, and Grafana, all deployed via Helm charts. Specifically, you:
- Configured Consul and Envoy to expose application metrics to Prometheus.
- Deployed Consul using the official helm chart.
- Deployed Prometheus and Grafana using their official Helm charts.
- Deployed a multi-tier demo application that was configured to be scraped by Prometheus.
- Started a traffic simulation deployment, and observed the metrics in Prometheus and Grafana.
Because all of these programs can run outside of Kubernetes, you can set this pipeline up in any environment or collect metrics from workloads running on mixed infrastructure.
To learn more about the configuration options in Consul that enable layer 7 metrics collection with or without Kubernetes, refer to our documentation. For more information on centrally configuring Consul, take a look at the centralized configuration documentation.
