Tempo HA docs - How to enable trace metrics
If your system is not monitored with metrics but generates traces, Charmed Tempo HA can be configured to generate metrics from them and forward the metrics to a Prometheus data source using the Prometheus remote write protocol.
We are going to assume that you have already deployed Charmed Tempo HA. See this doc for how to do that.
Also see:
Deploy a worker with the metrics-generator role
To enable trace metrics, you first need to ensure that your deployment has the metrics-generator role enabled. How to do that depends on your specific set-up.
Monolithic mode
If you intend to use Charmed Tempo HA in its monolithic mode, you will have a worker node with the role set to "all". This will implicitly also enable the metrics-generator role, so you don’t need to do anything else to enable it, and you can skip ahead to the next section.
Distributed mode
If you intend to use Charmed Tempo HA in its distributed mode, you will need to add a worker node with the metrics-generator role assigned.
For example:
juju deploy -m mymodel tempo-worker-k8s --config role-metrics-generator=true --config role-all=false tempo-metrics-generator
As Charmed Tempo HA workers only support a single role per application, don’t forget to unassign the "all" role, which is enabled by default!
Then relate it to the coordinator:
juju relate -m mymodel tempo tempo-metrics-generator
Integrate with a prometheus remote-write endpoint
At this point you will have a tempo application (an instance of the tempo-coordinator-k8s charm) related to a worker with the metrics-generator role enabled (or "all").
The tempo application will now set active status with a message warning you that Tempo is capable of generating metrics, but it isn’t related to a loki-push-api endpoint and therefore it isn’t generating any.
If you have deployed a dedicated worker with the metrics-generator role, that application will go to blocked until you relate the coordinator with a prometheus remote-write endpoint.
To solve this situation, integrate tempo with a prometheus_remote_write provider such as the prometheus-k8s charm. Assuming a prometheus is deployed in the same model as tempo:
juju relate -m mymodel tempo:send-remote-write prometheus:receive-remote-write
At this point, tempo should be sending trace-generated metrics to Prometheus and you should be able to find them in Grafana.
Last updated 2 months ago.