---
title: Charmhub | Deploy Canonical Observability Stack Lite using Charmhub - The Open
  Operator Collection
description: Deploy the latest version of Canonical Observability Stack Lite on any
  cloud.
url: https://charmhub.io/cos-lite/docs/how-to/troubleshoot-integrations
---

##### We've discontinued the registration of new Bundles

New Bundle registrations are no longer accepted. Existing bundles remain functional. We recommend using the Juju Terraform Provider for new deployments.

[Learn more](https://discourse.charmhub.io/t/discontinuing-new-charmhub-bundle-registrations/15344)

# Canonical Observability Stack Lite

[Canonical Observability](https://charmhub.io/publisher/observability "View all packages from Canonical Observability")
| bundle

* [Canonical Observability](https://charmhub.io/publisher/observability "View all packages from Canonical Observability")
  | bundle

Platform:

stable 26

```
juju deploy cos-lite
```

[Learn to deploy on juju >](https://juju.is/docs/juju/manage-applications)

---

#### Relevant links

* [Homepage](https://github.com/canonical/cos-lite-bundle)

---

#### Contacts

* [Submit a bug](https://github.com/canonical/cos-lite-bundle/issues)

---

Share your thoughts on this charm with the community on discourse.

[Join the discussion](https://discourse.charmhub.io/)

---

[Add docs to a charm](https://juju.is/docs/sdk/add-docs-to-your-charmhub-page)

Integrating a charm with [COS](https://charmhub.io/topics/canonical-observability-stack) means:

* having your app’s metrics and corresponding alert rules reach [prometheus](https://charmhub.io/prometheus-k8s/)
* having your app’s logs and corresponding alert rules reach [loki](https://charmhub.io/loki-k8s/)
* having your app’s dashboards reach [grafana](https://charmhub.io/grafana-k8s/)

The COS team is responsible for some aspects of testing, and some aspects of testing belong to the charms integrating with COS.

## [Tests for the built-in alert rules](https://charmhub.io/cos-lite/docs/how-to/troubleshoot-integrations#p-32235-tests-for-the-built-in-alert-rules)

### [Unit tests](https://charmhub.io/cos-lite/docs/how-to/troubleshoot-integrations#p-32235-unit-tests)

* You can use [`promtool test rules`](https://prometheus.io/docs/prometheus/latest/configuration/unit_testing_rules/) to make sure they fire when you expect them to fire. As part of the test you hard-code the time series values you are testing for.
* [`promtool check rules`](https://prometheus.io/docs/prometheus/latest/command-line/promtool/#promtool-check)
* [`cos-tool validate`](https://github.com/canonical/cos-tool). The advantage of cos-tool is that the same executable can validate both prometheus and loki rules.
* Make sure your alerts manifest matches the output of:
  + `juju ssh prometheus/0 curl localhost:9090/api/v1/rules | jq -r '.data.groups | .[] | .rules | .[] | .name'`.
  + `juju ssh loki/0 curl localhost:3100/loki/api/v1/rules`

### [Integration tests](https://charmhub.io/cos-lite/docs/how-to/troubleshoot-integrations#p-32235-integration-tests)

* A fresh deployment shouldn’t fire alerts, e.g. due to missing past data that is interpreted as 0.

## [Tests for the metrics endpoint and scrape job](https://charmhub.io/cos-lite/docs/how-to/troubleshoot-integrations#p-32235-tests-for-the-metrics-endpoint-and-scrape-job)

### [Integration tests](https://charmhub.io/cos-lite/docs/how-to/troubleshoot-integrations#p-32235-integration-tests)

* [`promtool check metrics`](https://prometheus.io/docs/prometheus/latest/command-line/promtool/#promtool-check) to lint the the metrics endpoint, e.g. `curl -s http://localhost:8080/metrics | promtool check metrics`.
* For scrape targets: when related to prometheus, and after a scrape interval elapses (default: 1m), all prometheus targets listed in `GET /api/v1/targets` should be `"health": "up"`. Repeat the test with/without ingress, TLS.
* For remote-write (and scrape targets): when related to prometheus, make sure that `GET /api/v1/labels` and `GET /api/v1/label/juju_unit` have your charm listed.
* Make sure that the metric names in your alert rules have matching metrics in your own `/metrics` endpoint.

## [Tests for log lines](https://charmhub.io/cos-lite/docs/how-to/troubleshoot-integrations#p-32235-tests-for-log-lines)

### [Integration tests](https://charmhub.io/cos-lite/docs/how-to/troubleshoot-integrations#p-32235-integration-tests)

* When related to loki, make sure your logging sources are listed in:
  + `GET /loki/api/v1/label/filename/values`
  + `GET /loki/api/v1/label/juju_unit/values`.

## [Tests for dashboards](https://charmhub.io/cos-lite/docs/how-to/troubleshoot-integrations#p-32235-tests-for-dashboards)

### [Unit tests](https://charmhub.io/cos-lite/docs/how-to/troubleshoot-integrations#p-32235-unit-tests)

* json lint

### [Integration tests](https://charmhub.io/cos-lite/docs/how-to/troubleshoot-integrations#p-32235-integration-tests)

* Make sure the dashboards manifest you have in the charm matches `juju ssh grafana/0 curl http://admin:password@localhost:3000/api/search`.

## [Remove duplications](https://charmhub.io/cos-lite/docs/how-to/troubleshoot-integrations#p-32235-remove-duplications)

### [Multiple grafana-agent apps related to the same principle](https://charmhub.io/cos-lite/docs/how-to/troubleshoot-integrations#p-32235-multiple-grafana-agent-apps-related-to-the-same-principle)

Charms should use `limit: 1` for the cos-agent relation ([example](https://github.com/canonical/zookeeper-operator/blob/main/metadata.yaml#L31)), but this cannot be enforced by grafana-agent itself. You can confirm this is the case with jq:

```
juju export-bundle | yq -o json '.' | jq -r '
  .applications as $apps |
  .relations as $relations |
  $apps
  | to_entries
  | map(select(.value.charm == "grafana-agent")) | map(.key) as $grafana_agents |
  $apps
  | to_entries
  | map(.key) as $valid_apps |
  $relations
  | map({
      app1: (.[0] | split(":")[0]),
      app2: (.[1] | split(":")[0])
    })
  | map(select(
      ((.app1 | IN($grafana_agents[])) and (.app2 | IN($valid_apps[]))) or
      ((.app2 | IN($grafana_agents[])) and (.app1 | IN($valid_apps[])))
    ))
  | map(if .app1 | IN($grafana_agents[]) then .app2 else .app1 end)
  | group_by(.)
  | map({app: .[0], count: length})
  | map(select(.count > 1))
'
```

If the same principal has more than one cos-agent relation, you would see output such as:

```
[
  {
    "app": "openstack-exporter",
    "count": 2
  }
]
```

Otherwise, you’d get:

```
jq: error (at <stdin>:19): Cannot iterate over null (null)
```

(which is good).

You can achieve this also using the status yaml. Save the following script to `is_multi_agent.py` and run with:

```
juju status --format=yaml | ./is_multi_agent.py
```

If there is a problem, you would see output such as:

```
openstack-exporter/19 is related to more than one grafana-agent subordinate: {'grafana-agent-container', 'grafana-agent-vm'}
```

```
#!/usr/bin/env python3

import yaml, sys

status = yaml.safe_load(sys.stdin.read())

# A mapping from grafana-agent app name to the list of apps it's subordiante to
agents = {
    k: v["subordinate-to"]
    for k, v in status["applications"].items()
    if v["charm"] == "grafana-agent"
}
# print(agents)

for agent, principals in agents.items():
    for p in principals:
        for name, unit in status["applications"][p].get("units", {}).items():
            subord_apps = {u.split("/", -1)[0] for u in unit["subordinates"].keys()}
            subord_agents = subord_apps & agents.keys()
            if len(subord_agents) > 1:
                print(
                    f"{name} is related to more than one grafana-agent subordinate: {subord_agents}"
                )
```

### [Grafana-agent related to multiple principles on the same machine](https://charmhub.io/cos-lite/docs/how-to/troubleshoot-integrations#p-32235-grafana-agent-related-to-multiple-principles-on-the-same-machine)

The grafana-agent vm charm can only be related to one principal in the same machine.

Save the following script to `is_multi.py` and run with:

```
juju status --format=yaml | ./is_multi.py
```

If there is a problem, you would see output such as:

```
ga is subordinate to both 'co', 'nc' in the same machines {'24'}
```

```
#!/usr/bin/env python3

import yaml, sys

status = yaml.safe_load(sys.stdin.read())

# A mapping from grafana-agent app name to the list of apps it's subordiante to
agents = {
    k: v["subordinate-to"]
    for k, v in status["applications"].items()
    if v["charm"] == "grafana-agent"
}

for agent, principals in agents.items():
    # A mapping from app name to machines
    machines = {
        p: [u["machine"] for u in status["applications"][p].get("units", {}).values()]
        for p in principals
    }

    from itertools import combinations

    for p1, p2 in combinations(principals, 2):
        if overlap := set(machines[p1]) & set(machines[p2]):
            print(
                f"{agent} is subordinate to both '{p1}', '{p2}' in the same machines {overlap}"
            )
```

## [Additional thoughts](https://charmhub.io/cos-lite/docs/how-to/troubleshoot-integrations#p-32235-additional-thoughts)

* A rock’s CI could dump a record of the `/metrics` endpoint each time the rock is built. This way some integration tests could turn into unit tests.

## [See also](https://charmhub.io/cos-lite/docs/how-to/troubleshoot-integrations#p-32235-see-also)

* [Prometheus-k8s docs - Troubleshooting integrations](https://discourse.charmhub.io/t/prometheus-k8s-docs-troubleshooting-integrations/14351)
* [Loki-k8s docs - Troubleshooting missing logs](https://discourse.charmhub.io/t/loki-k8s-docs-troubleshooting-missing-logs/14187)

---