Consul service mesh deploys an Envoy sidecar proxy alongside each service instance in a datacenter. The sidecar proxy brokers traffic between the local service instance and other services registered with Consul. The proxy is aware of all traffic that passes through it. In addition to securing inter-service communication, the proxy can also collect and expose data about the service instance. Starting with version 1.5, Consul service mesh is able to configure Envoy to expose layer 7 metrics, such as HTTP status codes or request latency, and as of 1.10 is able to export full mesh telemetry for Consul native resources as well as services including Consul agents and gateways.
In this tutorial, you will:
- Configure Consul to expose Envoy metrics to Prometheus.
- Deploy Prometheus and Grafana using their official Helm charts.
- Deploy Consul using the official helm chart.
- Deploy a multi-tier demo application that is configured to be scraped by Prometheus.
- Start a traffic simulation deployment, and observe the application traffic in Grafana.
Tip: While this tutorial shows you how to deploy a metrics pipeline on Kubernetes, all the technologies the tutorial uses are platform agnostic; Kubernetes is not necessary to collect and visualize layer 7 metrics with Consul service mesh.
»Prerequisites
If you already have a Kubernetes cluster running with Helm and kubectl installed, you can start on the tutorial right away. If not, set up a Kubernetes cluster using your favorite method that supports persistent volume claims, or install and start Minikube v1.10.1+ or kind v0.8.1+.
You must also install kubectl, and both install and initialize Helm. Also, to ensure you have the latest helm charts for Consul, Prometheus, and Grafana, you can run the following command.
Next, clone the GitHub repository that contains the files you'll use with this tutorial.
Change directories into the tutorial specific folder in the repository you just cloned.
Checkout the tagged version verified for this tutorial.
We'll refer to this directory as your working directory, and you'll run the rest of the commands in this tutorial from this directory.
»Deploy Consul service mesh using Helm
Once you have installed the prerequisites, you're ready to install Consul.
Open the file in your working directory called helm/consul-values.yaml and
review the configuration. It should match the YAML shown below.
Warning: By default, the chart will install an insecure configuration of Consul. This provides a less complicated out-of-box experience for new users, but is not appropriate for a production setup. Review the Secure Consul and Registered Services on Kubernetes tutorial for instructions on how to secure your datacenter for production.
Deploy Consul to your Kubernetes cluster.
The output will be a list of all the Kubernetes resources created. The output has been abbreviated, but will look similar to the following.
Check that Consul is running in your Kubernetes cluster using kubectl. Consul
setup is complete when all pods have a status of Running, as illustrated in the
following output.
»Deploy the metrics pipeline
Consul service mesh can integrate with a variety of metrics tooling,
but in this tutorial, you will use Prometheus and Grafana to collect and visualize
metrics. One important thing to be mindful of is that you do not want to inject
Envoy sidecars to your Prometheus and Grafana pods. You can disable this by
configuring podAnnotations with the "consul.hashicorp.com/connect-inject": "false"
annotation for each of those applications. Refer to their respect values files
in the helm directory for more information.
»Deploy Prometheus with Helm
Install the official Prometheus Helm chart using the values in
helm/prometheus-values.yaml.
Check that Prometheus is running in your Kubernetes cluster using kubectl.
Prometheus setup is complete when all pods have a status of Running, as illustrated
in the following output.
»Deploy Grafana with Helm
Installing Grafana will follow a similar process. Install the
official Grafana Helm chart using the values in
helm/grafana-values.yaml. This configuration will tell Grafana
to use Prometheus as a datasource, and set the admin password to
password.
Check that Grafana is running in your Kubernetes cluster using kubectl.
Grafana setup is complete when all pods have a status of Running, as illustrated
in the following output.
To expose the Grafana UI outside the cluster, issue the following command.
Leave this port-forward session active so that you can visit the UI again later once metrics are being collected.
Navigate to http://localhost:3000 in a browser tab and
log in to the Grafana UI using admin as the username and password as the password.
Once you have logged into the Grafana UI, hover over the dashboards icon (four
squares in the left-hand menu), and then click the Manage option.
This will take you to a page that gives you some choices about how to upload
Grafana dashboards. Click the Import button on the right-hand side of the
screen. Open the file in the grafana subdirectory named hashicups-dashboard.json
and copy the contents into the JSON window of the Grafana UI. Click through the
rest of the options, and you will end up with a dashboard waiting for data to display.
»Deploy a demo application on Kubernetes
Now that your monitoring pipeline is set up, deploy a demo application that will generate data. We will be using HashiCups, an application that emulates an online order app for a coffee shop. For this tutorial, the HashiCups application includes a React front end, a GraphQL API, a REST API and a Postgres database.
All the files defining HashiCups are in the hashicups directory. Open a new terminal,
and deploy the demo application.
Check that HashiCups is running in your Kubernetes cluster using kubectl.
HashiCups setup is complete when all pods have a status of Running, as illustrated
in the following output.
Test the application by viewing the React front end. You can do this by forwarding the frontend deployment's port 80 to your development host.
Navigate to http://localhost:8080 in a browser window. You should observe the following screen.
Now that the application is running let's verify that Consul did, in fact,
configure Envoy to publish metrics at port 20200. The Envoy side-car proxy
can be reached at port 19000. Enter CTRL-C to stop the port forward tunnel
for the frontend service on port 80, and then issue the following command which
will open a tunnel to the Envoy proxy.
Navigate to http://localhost:19000/config_dump in a browser window. You should
observe what looks like a raw JSON document dumped to the screen. This is the Envoy
configuration. Search for 20200 and you should find two different stanzas that
reference this port. One of them is included next for reference.
This confirms that Consul has configured Envoy to publish Prometheus metrics.
Enter CTRL-C to stop the port-forward session from the side-car proxy. You
will not need to reference it again for the remainder of the tutorial.
»Visualize application metrics
While Grafana is optimized for at a glance observability, the Prometheus UI can be useful as well. Issue the following command to expose the Prometheus UI to your development host.
»Discover available metrics with Prometheus
Navigate to http://localhost:9090 in a browser window, and you should observe the default
Prometheus UI. In the textbox at the top of the screen paste sum by(__name__)({app="product-api"})
and then click the button labeled Execute. Your screen will now look similar
to the following.
You have now performed a PromQL
query that will list all available metrics for resources that have the product-api label.
In this case, that is the REST API resource you deployed from the app folder. This list of metrics
can be used as constraints for further PromQL queries both here and in the Grafana UI.
Next, from within the Prometheus UI, click the dropdown menu item labeled Status in the menu bar at the top of the screen, and click on the option labeled Targets. This will navigate the UI to the Targets screen where you can review the resources that Prometheus is monitoring. The Prometheus Helm chart you installed earlier is configured to monitor the Kubernetes infrastructure as well as Prometheus itself.
Click on the button labeled show less next to each section header except the one labeled kubernetes-pods. Your screen should appear similar to the following.
Notice that several pods are running, and have a State of UP. Look at the first label
for each pod in the Label column. You should have entries for app="frontend", app="postgres",
app="products-api", and app="public-api". This confirms that Prometheus is collecting
metrics from all of your annotated resources, and that they are all up and running.
Go back to the terminal where your port-forward session for the Prometheus UI is running,
and type CTRL-C to end the session. You will not need to access the Prometheus UI for
the remainder of the tutorial.
»Simulate traffic
Now that you know the application is running, start generating some load so that you will have some metrics to look at in Grafana.
Envoy exposes a huge number of metrics. Which metrics are important to monitor will depend on your application. For this tutorial we have preconfigured a HashiCups-specific Grafana dashboard with a couple of basic metrics, but you should systematically consider what others you will need to collect as you move from testing into production.
»View Grafana dashboard
Now that you have metrics flowing through your pipeline, and a traffic simulation deployment running, navigate back to your Grafana dashboard in the Grafana UI, and you should observe a screen similar to the following.
Notice that once you started the traffic simulation deployment Prometheus started to log active connections. This dashboard is simplistic, but illustrates that metrics are flowing through the pipeline, and should give you a reasonable starting point for setting up your own observability tooling.
»Viewing traffic in Consul UI
You can also view service metrics in the Consul UI. In a new terminal session, issue the following command to expose the Consul UI to the development host.
Open localhost:8500 in a new browser tab, and navigate
to the Services screen. Select the postgres service. You should observe that
a chart with some basic metrics is embedded in the service tile.
Hover over the four tile elements to get tooltip descriptions of the different metrics.
You can also hover the timeline chart embedded in the tile, and review additional metrics for any point in time during the last 15 minutes.
»Clean up
If you want to get rid of the configuration files and Consul Helm chart, recursively
remove the learn-consul-kubernetes directory.
`
»Next steps
In this tutorial, you set up layer 7 metrics collection and visualization in a Kubernetes cluster using Consul service mesh, Prometheus, and Grafana, all deployed via Helm charts. Specifically, you:
- Configured Consul and Envoy to expose application metrics to Prometheus.
- Deployed Consul using the official helm chart.
- Deployed Prometheus and Grafana using their official Helm charts.
- Deployed a multi-tier demo application that was configured to be scraped by Prometheus.
- Started a traffic simulation deployment, and observed the metrics in Prometheus and Grafana.
Because all of these programs can run outside of Kubernetes, you can set this pipeline up in any environment or collect metrics from workloads running on mixed infrastructure.
To learn more about the configuration options in Consul that enable layer 7 metrics collection with or without Kubernetes, refer to our documentation. For more information on centrally configuring Consul, take a look at the centralized configuration documentation.
