HashiCorp Diagnostics (hcdiag) is a troubleshooting data-gathering tool that
you can use to collect and archive important data from Consul server
environments. The information gathered by hcdiag is well-suited for sharing
with teams during incident response and troubleshooting.
In this tutorial, you will:
- Retrieve example repository with Git
- Create a local Consul datacenter using Docker Compose
- Use the created environment to execute basic
hcdiagcommands - Explore the contents of files created by the tool
- Learn about additional
hcdiagoptions and how to use a configuration file withhcdiag
Note: The hcdiag tool is currently available only for Consul on Linux.
»Prerequisites
»Docker
You will need a local install of Docker running on your machine for this tutorial. You can find the instructions for installing Docker here. Docker Engine 20.10.7 was used in this tutorial.
»Docker Compose
Docker Compose is a tool for defining and running multi-container Docker applications. With Compose, you use a YAML file to configure your application’s services. Visit the Docker Compose install guide for operating system specific installation instructions. Docker Compose format 3.7 was used in this tutorial.
»Git
Git is a version control system used for interacting with code repositories. Visit the Git download page for installation instructions.
»Clone GitHub repository
Clone the GitHub repository containing the configuration files and resources.
- HTTPS
- SSH
Change into the directory with the newly cloned repository.
Check out the v0.2 tag of the repository.
Change into the directory that contains the complete configuration files for this tutorial.
»Create the Consul datacenter
From within your working directory, run the following Docker Compose command to create local Consul datacenter featuring three Consul servers and a single client.
Note: Your first run will take the longest as Docker will first pull the respective images from the Docker Hub repository. Additional runs will not require to download the image again and should only take a few seconds to complete.
»Access the Consul client
Open an interactive shell to the Consul server container.
Note that your terminal prompt will now appear differently, like this:
Set the required environment variables that enable hcdiag to communicate with
your secure Consul cluster.
Run the consul members command to ensure the Consul datacenter is up and running.
»Install necessary tools
Before you can download hcdiag, you need to install the curl and unzip tools. The Consul container is based on Alpine Linux, so you can do this with the apk package manager.
First update package sources.
Then install curl and unzip.
»Download and use hcdiag
You are now ready to download hcdiag from the HashiCorp releases repository. Recall that hcdiag is currently only available as a Linux binary. hcdiag also not currently available as a Linux package (such as apt or apk).
Download the binary with curl.
Unzip and remove the archive.
Move the hcdiag executable into /usr/sbin so that you can invoke the command using just its name.
Run hcdiag with the consul flag. This will gather your available environment and Consul product information.
Tip: You can also invoke hcdiag without options to gather all available environment and product information. To learn about all executable options, run hcdiag -h.
»Examine the output
List the directory for tar+gzip archive files to discover the file that hcdiag created.
The filename contains a support prefix, followed by a timestamp. You can unpack the archive and further examine its contents.
The archive extracts three files into a temporary directory:
Manifest.jsoncontains information describing the hcdiag run, including configuration options used, run duration, and a count of any errors encountered.Results.jsoncontains information about the environment and the output from an invocation ofConsul debug.ConsulDebug.tar.gzcontains the output from invokingConsul debugas described in
You can further explore the ConsulDebug.tar.gz archive by unpacking it and examining its contents.
Feel free to inspect the bundle to ensure it contains only information that is appropriate to share based on your use-case or situation. The tool will not obscure secrets or sensitive information.
Consul Enterprise users: If you are a Consul Enterprise user, you can share the output from hcdiag with HashiCorp Customer Success to greatly reduce the amount of information gathering needed in a support request.
The tool only works locally and does not export or share the diagnostic bundle with anyone. You must use other tools to transfer it to a secure location so you can share it with specific support staff who need to view it.
»Configuration file
You can configure hcdiag behavior with a HashiCorp Configuration Language (HCL) formatted file. If you examine the Results.json file under the respective product key, such as Consul, you can find the commands that are executed.
Here is a minimal configuration file that instructs hcdiag to exclude the consul debug command shown as an example.
If you create this file as diag.hcl and execute hcdiag as follows, then you could expect the output to resemble the example output.
You do not have to create the configuration and execute the command again, but if you compare the example output to the previous invocation that you ran, you notice that the excluded Consul agent metrics information is not present.
»Production usage tips
By default, the hcdiag tool includes files for up to 72 hours back from the current time. You can specify the desired time range using the -include-since flag.
If you are concerned about impacting performance of your Consul servers, you can specify that the seekers to not run concurrently, and instead be invoked serially with the -serial flag.
Deploying hcdiag in production involves a workflow similar to the following:
Place the
hcdiagbinary on the Consul system in scope - this could be a Consul server or a Consul client.When running with a configuration file and the
-configflag, ensure that the specified configuration file is readable by the user that executeshcdiag.Ensure that the current directory or that specified by the
destflag is writable by the user that executeshcdiag.Ensure connectivity to the HashiCorp products that
hcdiagneeds to connect to during the run. Export any required environment variables for establishing connection or passing authentication tokens as necessary.Decide on a duration for information gathering, noting that the default is to gather for up to 72 hours back in server log output. Adjust your needs as necessary with the
-include-sinceflag. For example, to include only 24 hours of log output, invoke as:Limit what is gathered with the
-includesflag. For example,-includes /var/log/consul-*,/var/log/nomad-*instructshcdiagto only gather logs matching the specified Consul and Nomad filename patterns.Use the
-dryrunflag to observe what hcdiag will do without anything actually being done for testing configuration and options.
»Clean up your environment
To clean up from exploring hcdiag, first exit the container.
Stop and remove all containers.
»Summary
In this tutorial, you retrieved a Git repository, created a local Consul datacenter with Docker Compose, and used the local environment to explore the hcdiag tool in the context of gathering information from a running Consul environment.
You also learned about the available configuration flags, the configuration file, and production specific tips for using hcdiag.
»Help and Reference
Feel free to explore the following resources for additional help with troubleshooting your Consul environment.
