Tempo HA docs - How to enable tracing in Istio

If you have an Istio deployment in your cluster and have some charmed workloads that are on the mesh, you can configure Istio to enable it to auto generate traces from traffic between your charmed workloads inside the mesh and export them to a tracing backend.

Deploy Istio

First, we need to deploy istio-k8s in an istio-system model.

This would deploy Istio in its ambient mode.

See more: Istio’s ambient mode

juju add-model istio-system

juju deploy istio-k8s istio --channel edge --trust --config auto-allow-waypoint-policy=false

auto-allow-waypoint-policy=false is needed when deploying Istio to allow waypoints to communicate with their workloads.

Add COS to the mesh

The next step is to add some workloads to the mesh and have some traffic flowing inside your mesh.

For this example, we’ll be deploying cos-lite alongside Tempo HA in an example model mesh and add them to Istio’s mesh.

Deploy `cos-lite`

juju add-model mesh
juju deploy cos-lite --trust --channel edge

Deploy Tempo HA

Check this guide to deploy Tempo HA.

Then run jhack imatrix fill to cross relate everything in the model.

You can also relate the charms separately using juju integrate.

Deploy `istio-beacon-k8s`

istio-beacon-k8s charm will add charmed workloads deployed on the same model to the mesh. It would also deploy a waypoint proxy, an ambient-mode Istio component that will be responsible for generating and exporting mesh traces.

Integrate `istio` with `tracing`

In the mesh model,

juju offer tempo:tracing

In the istio-system model,

juju consume admin/mesh.tempo

Integrate charm-tracing and workload-tracing

juju integrate istio-k8s:charm-tracing tempo
juju integrate istio-k8s:workload-tracing tempo

Until https://github.com/canonical/istio-k8s-operator/issues/30 is fixed, you’d need to apply this kubernetes manifest

microk8s.kubectl apply -f - <<EOF
apiVersion: networking.istio.io/v1alpha3
kind: ServiceEntry
metadata:
  name: tempo
spec:
  hosts:
  - tempo-0.tempo-endpoints.mesh.svc.cluster.local
  ports:
  - number: 4317
    name: grpc-otel
    protocol: GRPC
  resolution: DNS
EOF

Visualize traces in Grafana

Open Grafana web UI form your local web browser and login using the admin password.

Then, Toggle Menu → Explore → Choose your Tempo datasource

This should result in an output similar to the one below:

As expected in Istio’s ambient mode, mesh traces are generated in the L7 layer by the waypoint proxy (i.e istio-beacon-k8s), so you’d find traces generated under istio-beacon-k8s-mesh-waypoint.mesh service name.

Stitching spans together

By design, Istio will not propagate trace headers, so it would be the workload’s responsibility to propagate the trace headers.

Charmed Istio will generate and export traces using OpenTelemetry protocol and therefore, if a charmed workload on the mesh is already instrumented to propagate OpenTelemetry headers, Tempo would stitch together spans generated from that workload with the one generated by its waypoint proxy along with any other workloads that are instrumented in the same manner as you can see in the screenshot above.

Last updated 26 days ago.

Help improve this document in the forum.