How to: Testing COS integrations

Integrating a charm with COS means:

The COS team is responsible for some aspects of testing, and some aspects of testing belong to the charms integrating with COS.

Tests for the built-in alert rules

You can use promtool test rules to make sure they fire when you expect them to fire. As part of the test you hard-code the time series values you are testing for.
promtool check rules
cos-tool validate. The advantage of cos-tool is that the same executable can validate both prometheus and loki rules.
Make sure your alerts manifest matches the output of:
- juju ssh prometheus/0 curl localhost:9090/api/v1/rules | jq -r '.data.groups | .[] | .rules | .[] | .name'.
- juju ssh loki/0 curl localhost:3100/loki/api/v1/rules

A fresh deployment shouldn’t fire alerts, e.g. due to missing past data that is interpreted as 0.

promtool check metrics to lint the the metrics endpoint, e.g. curl -s http://localhost:8080/metrics | promtool check metrics.
For scrape targets: when related to prometheus, and after a scrape interval elapses (default: 1m), all prometheus targets listed in GET /api/v1/targets should be "health": "up". Repeat the test with/without ingress, TLS.
For remote-write (and scrape targets): when related to prometheus, make sure that GET /api/v1/labels and GET /api/v1/label/juju_unit have your charm listed.
Make sure that the metric names in your alert rules have matching metrics in your own /metrics endpoint.

When related to loki, make sure your logging sources are listed in:
- GET /loki/api/v1/label/filename/values
- GET /loki/api/v1/label/juju_unit/values.

Make sure the dashboards manifest you have in the charm matches juju ssh grafana/0 curl http://admin:password@localhost:3000/api/search.

A rock’s CI could dump a record of the /metrics endpoint each time the rock is built. This way some integration tests could turn into unit tests.