COS-Lite docs - Add distributed tracing support with Tempo

Table of Contents

If you have an existing COS-Lite deployment and you wish to add an integration to the Tempo distributed tracing backend, follow these steps.

Deploy tempo-coordinator-k8s

In the same juju model as you have COS-Lite deployed, deploy the tempo-coordinator-k8s app.

juju deploy tempo-coordinator-k8s --channel edge tempo --trust

We recommend naming your tempo-coordinator app "tempo" as that is the single entry point to the tempo deployment as a whole: you will never have to interact with the worker nodes directly.

Deploy tempo-worker-k8s

juju deploy tempo-worker-k8s --channel edge tempo-worker --trust

In this tutorial we deploy the ‘monolithic’ version of tempo, where a single worker node is assigned all the roles. For alternative deployment modes and a migration guide, refer to this other doc.

Integrate with s3

Tempo uses object storage for storing traces and the charm consequently requires an s3 integration.

If you don’t have an s3 bucket ready at hand, follow this guide to deploy minio in your testing environment.

Once you’re done deploying minio and s3, you can run:

juju integrate tempo s3

And wait for tempo to go to active/idle.

Integrate coordinator and workers

juju integrate tempo tempo-worker

At this point your juju status should look like this:

image

Coordinator is reporting ‘degraded’ because not all roles are assigned in the recommended number (see this open bug for progress on making the status message more informative).

Integrate with COS-Lite

You can enable self-monitoring for tempo by integrating it with the other COS-Lite components.

juju integrate loki:logging tempo:logging juju integrate s3:s3-credentials tempo:s3
juju integrate tempo:grafana-dashboard grafana:grafana-dashboard
juju integrate tempo:grafana-source grafana:grafana-source
juju integrate tempo:metrics-endpoint prometheus:metrics-endpoint
juju integrate tempo:tempo-cluster tempo-worker:tempo-cluster
juju integrate traefik:traefik-route tempo:ingress

Similarly, you can enable tracing in COS-Lite by integrating the COS-Lite charms that support it to tempo over tracing:

juju integrate tempo:tracing alertmanager:tracing
juju integrate tempo:tracing catalogue:tracing
juju integrate tempo:tracing grafana:tracing
juju integrate tempo:tracing loki:tracing
juju integrate tempo:tracing prometheus:tracing
juju integrate tempo:tracing traefik:tracing

You can also achieve the same by running jhack imatrix fill.

Integrate with a CA

If you have a charm offering a certificates endpoint such as self-signed-certificates, you can integrate it with tempo:

juju integrate tempo:certificates ca:certificates

to enable traces to be sent to tempo over https (or grpcs).

Note that for this to work, tempo needs to trust the same CA that traefik trusts. If you’re using different CAs to provide certificates to tempo and traefik, you’ll need to integrate the CA charms with cert-transfer.

Last updated 2 months ago.

Help improve this document in the forum.