Prometheus

  • By Balbir Thomas
Channel Version Revision Published Runs on
latest/stable 79 79 21 Oct 2022
Ubuntu 20.04
latest/candidate 79 79 21 Oct 2022
Ubuntu 20.04
latest/beta 79 79 21 Oct 2022
Ubuntu 20.04
latest/edge 86 86 Yesterday
Ubuntu 20.04
1.0/stable 79 79 21 Oct 2022
Ubuntu 20.04
1.0/candidate 79 79 21 Oct 2022
Ubuntu 20.04
1.0/beta 79 79 21 Oct 2022
Ubuntu 20.04
1.0/edge 79 79 21 Oct 2022
Ubuntu 20.04
juju deploy prometheus-k8s
Show information

Platform:

Prometheus k8s Charm Relations

At present the Prometheus Charmed Operator for Kubernetes supports seven relations.

Requires

Metrics Endpoint:

  metrics-endpoint:
    interface: prometheus_scrape

Charms may forward information about their metrics endpoints and associated alert rules to the Prometheus charm over the metrics-endpoint relation using the prometheus_scrape interface. In order for these metrics to be aggregated by this Prometheus charm all that is required is to relate the two charms as in:

juju relate kube-state-metrics-k8s prometheus-k8s

Charms that seek to provide metrics endpoints and alert rules for Prometheus must do so using the provided prometheus_scrape charm library. This library by implementing the metrics-endpoint relation, not only ensures that scrape jobs and alert rules are forward to Prometheus but also that these are updated any time the metrics provider charm is upgraded. For example new alert rules may be added or old ones removed by updating and releasing a new version of the metrics provider charm. While it is safe to update alert rules as desired, care must be taken when updating scrape job specifications as this has the potential to break the continuity of the scraped metrics time series. In particular changing the following keys in the scrape job can break time series continuity

  • job_name
  • relabel_configs
  • metrics_relabel_configs
  • Any label set by static_configs

Evaluation of alert rules forwarded through the prometheus_scrape interface are automatically limited to the charmed application that provided these rules. This ensures that alert rule evaluation is scoped down to the charm providing the rules.

Alerting

  alertmanager:
    interface: alertmanager_dispatch

The Alertmanager Charm aggregates, deduplicates, groups and routes alerts to selected “receivers”. Alertmanager receives its alerts from Prometheus and this interaction is set up and configured using the alertmanager relation through the alertmanager_dispatch interface. Over this relation the Alertmanager charm keeps Prometheus informed of all Alertmanager instances (units) to which alerts must be forwarded. If your charm sets any alert rules then almost always it would need a relation with an Alertmanager charm which had been configured to forward alerts to specific receivers. In the absence of such a relation alerts even when raised will only be visible in the Prometheus user interface. A prudent approach to setting up an Observability stack is to do so in a manner such that it draws your attention to alarms as and when they are raised, without you having to periodically check a dashboard.

Ingress

  ingress:
    interface: ingress_per_unit
    limit: 1

Interactions with the Prometheus charm can not be assumed to originate within the same Juju model, let alone the same Kubernetes cluster, or even the same Juju cloud. Hence the Prometheus charm also supports an Ingress relation. There are multiple use cases that require an ingress, in particular

  • Using the Prometheus remote write endpoint across network boundaries.
  • Querying the Prometheus HTTP API endpoint across network boundaries.
  • Self monitoring of Prometheus that must happen across network boundaries to ensure robustness of self monitoring.
  • Supporting the Loki push API.
  • Exposing the Prometheus remote write endpoint to Grafana agent.

Prometheus typical needs a “per unit” Ingress. This per unit ingress is necessary since Prometheus exposes a remote write endpoint on a per unit basis. A per unit ingress relation is available in the traefik-k8s charm and this Prometheus charm does support that relation over ingress_per_unit interface.

Provides

Grafana Source

  grafana-source:
    interface: grafana_datasource

The Grafana Charm provides a data visualization solution for metrics aggregated by Prometheus and supports the creation of bespoke dashboards for such visualization. Grafana requires a data source for its dashboards and this Prometheus charm provides the data source through the grafana-source relation using the grafana_datasource interface. To visualize your charms metrics using Grafana the following steps are required

  • Add a relation between your charm (say cassandra-k8s) and Prometheus so that Prometheus can aggregate the metrics.
  • Add a relation between the Grafana and Prometheus charm so that metrics are forwarded to Grafana.
  • Add a relation between your charm and Grafana so that your charm can forward dashboards for its metrics to Grafana.

For example

juju relate cassandra-k8s prometheus-k8s
juju relate prometheus-k8s grafana-k8s
juju relate cassandra-k8s grafana-k8s

Remote Write

  receive-remote-write:
    interface: prometheus_remote_write

Metrics may also be pushed to this Prometheus charm through the receive-remote-write relation using the prometheus_remote_write interface, which can be used with the Grafana Agent Charm to have metrics scraped by the Grafana Agent sent over to Prometheus.

Self metrics endpoint

self-metrics-endpoint:
    interface: prometheus_scrape

This Prometheus charm may forward information about its metrics endpoint and associated alert rules to another Prometheus charm over the self-metrics-endpoint relation using the prometheus_scrape interface. In order for these metrics to be aggregated by the remote Prometheus charm all that is required is to relate the two charms as in:

juju relate \
    prometheus-k8s:self-metrics-endpoint \
    remote-prometheus-charm:metrics-endpoint

Grafana dashboard

  grafana-dashboard:
    interface: grafana_dashboard

Over the grafana-dashboard relation using the grafana-dashboard interface, this Prometheus charm also provides meaningful dashboards about its metrics to be shown in a Grafana Charm .

In order to add these dashboards to Grafana all that is required is to relate the two charms in the following way:

juju relate \
    prometheus-k8s:grafana-dashboard \
    grafana-k8s:grafana-dashboard

Help us improve this documentation

Most of this documentation can be collaboratively discussed and changed on the respective topic in the doc category of the Charmhub forum. See the documentation guidelines if you’d like to contribute.

Last updated 3 months ago. Help improve this document in the forum.