In this tutorial, you will configure Consul service mesh to install a pre-configured instance of both Prometheus and Grafana suitable for development and evaluation purposes.
Consul service mesh on Kubernetes provides a deep integration with Prometheus, and even includes a built-in starter experience for installing Prometheus and Grafana to demo or for dev environments. The following service mesh resource metrics can be configured for immediate collection using the official Consul Helm chart.
- Envoy proxies
- Consul server and client agents
- Registered services
- Consul gateways
Note: This is a technology preview, and requires you to run an alpha version of Consul. This is not suitable for production environments.
To try the features in this tutorial, you will need a Kubernetes cluster with Consul service mesh installed. In case you do not have one, and are interested in testing the use cases in this tutorial, you can follow the Understand Consul service mesh tutorial to deploy a local Kubernetes cluster.
To install Consul service mesh on Kubernetes using the official Helm chart, you
must ensure you have enabled Consul service mesh features by setting the
connectInject.enabled flag to
true in the values file you supply to the Consul
Helm chart installation. When you enable the Consul service mesh, Consul injects
envoy-sidecar container into in each service deployment
you add to the mesh. The
consul-sidecar container is capable of coordinating
Layer 7 (L7) metrics collection (HTTP status codes, request latency, transaction
volume, etc.) for your service mesh applications. The metrics integration also supports collecting Consul agent metrics and gateway metrics. With a small amount of configuration,
this data can be exported to Prometheus and visualized in either Grafana
or the Consul UI. We've provided a sample configuration in a repository for you.
Note: If you followed the instructions in the Deploy Consul Service Mesh on Kubernetes tutorial, you will need to upgrade your mesh with the configuration discussed in this section.
Clone the GitHub repository that contains the files you'll use with this tutorial.
$ git clone https://github.com/hashicorp/learn-consul-kubernetes.git
Change directories into sample code for this tutorial located in the repository you just cloned.
$ cd learn-consul-kubernetes/service-mesh/observability
Checkout the tagged version verified for this tutorial.
git checkout tags/v0.0.7
Below is the minimal Helm configuration we've provided in your cloned repository. This configuration will enable both the Consul service mesh and its observability features.
global: name: consul datacenter: dc1 image: hashicorp/consul:1.10.0-alpha metrics: # Configures the Helm chart’s components to expose Prometheus metrics for the Consul service mesh. enabled: true # Publishes consul agent metrics. Only applicable if global.metrics.enabled is true. enableAgentMetrics: true server: replicas: 1 ui: enabled: true connectInject: enabled: true default: true # Configures metrics for Consul service mesh services. All values are overridable # via annotations on a per-pod basis. metrics: # Configures the Consul sidecar to run a merged metrics server # to combine and serve both Envoy and Application metrics. defaultEnableMerging: true controller: enabled: true # prometheus configures a Prometheus installation. prometheus: # When true, the Helm chart will install a demo Prometheus server # instance alongside Consul. enabled: true # grafana configures a Grafana installation. grafana: # When true, the Helm chart will install a demo grafana instance # alongside Consul. enabled: true
helm install to install Consul service mesh with the configuration
$ helm install -f config.yaml consul hashicorp/consul --version "0.31.1" NAME: consul ...TRUNCATED... $ helm status consul $ helm get all consul
Notice that there are several metrics settings included throughout the configuration. For the remainder of this tutorial, we'll discuss these settings and how they affect the observability configuration of your service mesh.
For a complete list of all possible settings, refer to the official documentation.
Consul's metrics features can either be enabled globally or per component. By
default, metrics features are disabled globally, and both the Consul UI and
service mesh are configured to inherit their enablement from the
global: name: consul datacenter: dc1 image: hashicorp/consul:1.10.0-alpha metrics: # Configures the Helm chart’s components to expose Prometheus metrics for the Consul service mesh. enabled: true # Publishes consul agent metrics. Only applicable if global.metrics.enabled is true. enableAgentMetrics: true
true as shown in this example YAML, you
are turning on metrics collection for Envoy proxies, registered services, and
Consul gateways globally. The configuration above also sets
true, which configures Consul to publish and collect metrics for all Consul
server and client agents.
Note: Consul Prometheus does not currently support TLS configurations. This
means that if
global.tls.httpsOnly are set to
and Consul agent metrics are enabled, the installation will error out during Helm
template rendering. You can work around this by setting the Helm chart value
global.metrics.enableGatewayMetrics, setting which is not shown above, is
turned on by default when metrics are enabled globally. You can override this
setting manually if you do not wish to collect metrics about your Consul gateways.
»Enable the Consul UI
To support visualizing metrics using the Consul UI, you must ensure that the Consul UI is enabled. We recommend you enable the Consul UI whether you plan to enable metrics or not.
ui: enabled: true
»Service mesh configuration
Consul service mesh is configured using settings contained within the
stanza. Set the
connectInject.enabled setting to
true to enable Consul service
connectInject: enabled: true default: true # Configures metrics for Consul service mesh services. All values are overridable # via annotations on a per-pod basis. metrics: # Configures the Consul sidecar to run a merged metrics server # to combine and serve both Envoy and Application metrics. defaultEnableMerging: true
true, as shown in this sample YAML, sidecars
will be automatically injected with your service deployment by default. Also, note
that a metrics stanza has been added for you, and that the
entry has been set to
true. This ensures that the
consul-sidecar that gets
injected for each deployment is configured to merge Envoy metrics with your
registered service metrics into a single metrics endpoint that Prometheus will
»Enable Prometheus and Grafana
To install a demo instance of both Prometheus and Grafana to your mesh using
their respective Helm charts, set the
true. With this feature enabled, you can test Consul with Prometheus
and Grafana natively integrated beginning on Day 0. However, this feature is not
intended for production environments. You should be aware that with this configuration,
upgrading to future versions of the Consul Helm chart may result in Prometheus
and Grafana being upgraded as well.
# prometheus configures a Prometheus installation. prometheus: # When true, the Helm chart will install a demo Prometheus server # instance alongside Consul. enabled: true # grafana configures a Grafana installation. grafana: # When true, the Helm chart will install a demo grafana instance # alongside Consul. enabled: true
»View in Consul UI
To view metrics in the UI, you will need to have services in the mesh
deployed to your datacenter. We have provided a demo application in the repo that
you downloaded earlier that you can use with this tutorial to test Consul's metrics
features. You can use
kubectl to deploy the demo application.
$ kubectl apply -f hashicups service/frontend-service created configmap/nginx-configmap created deployment.apps/frontend created service/products-api-service created serviceaccount/products-api created configmap/db-configmap created deployment.apps/products-api created service/postgres created deployment.apps/postgres created service/public-api-service created deployment.apps/public-api created
Ensure that the demo application is healthy with
kubectl get pods
to verify that all pods have a status of
Running before proceeding to the next
$ kubectl get pods NAME READY STATUS RESTARTS AGE consul-6kjbk 1/1 Running 0 3h42m consul-connect-injector-webhook-deployment-659cc98d77-vjjg5 1/1 Running 0 3h42m consul-controller-5788b8f6c7-65psj 1/1 Running 0 3h42m consul-server-0 1/1 Running 0 3h42m consul-webhook-cert-manager-5745cbb9d-94tdc 1/1 Running 0 3h42m frontend-6f548678bb-c277q 3/3 Running 0 35m grafana-76c87b68d8-fcfk5 1/1 Running 0 3h42m postgres-7847fc4696-xg6rj 3/3 Running 0 3h29m product-api-7f45c88999-fndxm 3/3 Running 0 40m prometheus-server-c5f9465d-wtlft 2/2 Running 0 3h42m public-api-7bb69fcfd6-6sldc 3/3 Running 0 35m
Once all application pods are running, start the traffic simulation job.
$ kubectl apply -f traffic.yaml configmap/k6-configmap created deployment.apps/traffic created
You can now view service metrics in the Consul UI. In a new terminal session, issue the following command to expose the Consul UI to your development host.
$ kubectl port-forward consul-server-0 8500:8500 Forwarding from 127.0.0.1:8500 -> 8500 Forwarding from [::1]:8500 -> 8500
localhost:8500 in a new browser tab, and navigate
to the "Services" screen. Select a service. You should observe that
a chart with some basic metrics is embedded in the service tile.
Hover over the four tile elements to get tooltip descriptions of the different metrics. You can also hover the timeline chart embedded in the tile, and review additional metrics for any point in time during the last 15 minutes.
In this tutorial, you were introduced to some observability features of Consul service mesh on Kubernetes, and were instructed on how to configure a demo installation of both Prometheus and Grafana suitable for development or evaluation. For more information on how to create a production grade observability pipeline for your Consul service mesh using Prometheus and Grafana in Kubernetes, refer to our Layer 7 Observability with Consul Service Mesh, Prometheus, Grafana, and Kubernetes. tutorial.
In addition to configuring metrics defaults globally via the Helm chart, you can
also provide, and even override, configuration on a per-pod basis using annotations.
Annotations are inspected by the
connect-inject handler, and if relevant, passed
on to the
consul-sidecar. Refer to the official documentation
for a complete discussion on how annotations will affect your metrics configuration.
In the next tutorial, Manage Traffic with Consul Service Mesh, you will be introduced to the comprehensive traffic management features that Consul provides at the proxy, service, and WAN levels.