- By Canonical Observability
|latest/stable||103||31 Jan 2023|
|latest/candidate||103||31 Jan 2023|
|latest/beta||103||31 Jan 2023|
|1.0/stable||103||31 Jan 2023|
|1.0/candidate||103||31 Jan 2023|
|1.0/beta||103||31 Jan 2023|
|1.0/edge||103||31 Jan 2023|
juju deploy prometheus-k8s
You will need Juju 2.9 to be able to run this command. Learn how to upgrade to Juju 2.9.
Deploy Kubernetes operators easily with Juju, the Universal Operator Lifecycle Manager. Need a Kubernetes cluster? Install MicroK8s to create a full CNCF-certified Kubernetes system in under 60 seconds.
Prometheus Charmed Operator for Kubernetes
The Prometheus Charmed Operator for Juju provides a monitoring solution using Prometheus, which is an open-source monitoring system and alerting toolkit. Besides it handles instantiation, scaling, configuration, and Day 2 operations specific to Prometheus.
This Charm is also part of the Canonical Observability Stack and deploys only the monitoring component of Prometheus in a Kubernetes cluster. An alerting service for Prometheus is offered through a separate Charm.
Create a Juju model for your operator, say "observability"
juju add-model observability
The Prometheus Charmed Operator may now be deployed using the Juju command line as in
juju deploy prometheus-k8s --channel=stable
Checking deployment status
Progress of the Prometheus charm deployment and its current status may be viewed anytime using the Juju command line.
juju status --color --relations
Once the Prometheus charm deployments completes, you may expect to see a status result such as
$ juju status --relations Model Controller Cloud/Region Version SLA Timestamp cos charm-dev-batteries microk8s/localhost 3.0.2 unsupported 14:27:58-03:00 App Version Status Scale Charm Channel Rev Address Exposed Message prometheus-k8s 2.33.5 active 1 prometheus-k8s stable 79 10.152.183.227 no Unit Workload Agent Address Ports Message prometheus-k8s/0* active idle 10.1.36.90 Relation provider Requirer Interface Type Message prometheus-k8s:prometheus-peers prometheus-k8s:prometheus-peers prometheus_peers peer
Accessing the Prometheus User Interface
Prometheus provides a user interface that let us explore the data that has collected such as metrics and associated alerts.
This UI is accessed on port 9090 at the Prometheus charm's address. This assumes the Prometheus host address is accessible from the host your browser is running on. For example the Juju status message shown above is from a
microk8s cluster, hence navigating to
http://10.152.183.227:9090 will show us the Prometheus UI.
Adding scrape targets
When Prometheus is deployed it will only scrape metrics from itself. This is the default behavior of Prometheus. Additional metrics endpoints may be added to the Prometheus charm through relations with other charms. Currently the following charms are supported
- Any charm that uses the
prometheus_scrapeinterface to provide a metrics endpoint and optionally alert rules for Prometheus.
- The Prometheus Scrape Target charm may be used to scrape metrics endpoint that are not part of any Juju Model.
- The Prometheus Scrape Config charm may be used to scrape metrics endpoints across different Juju models. The charm also support overriding some of the scrape job configurations provided by metrics endpoints.
At present the Prometheus Charmed Operator for Kubernetes supports eight relations.
metrics-endpoint: interface: prometheus_scrape
Charms may forward information about their metrics endpoints and associated alert rules to the Prometheus charm over the
metrics-endpoint relation using the
prometheus_scrape interface. In order for these metrics to be aggregated by this Prometheus charm all that is required is to relate the two charms as in:
juju relate kube-state-metrics-k8s prometheus-k8s
Charms that seek to provide metrics endpoints and alert rules for Prometheus must do so using the provided
prometheus_scrape charm library. This library by implementing the
metrics-endpoint relation, not only ensures that scrape jobs and alert rules are forward to Prometheus but also that these are updated any time the metrics provider charm is upgraded. For example new alert rules may be added or old ones removed by updating and releasing a new version of the metrics provider charm. While it is safe to update alert rules as desired, care must be taken when updating scrape job specifications as this has the potential to break the continuity of the scraped metrics time series. In particular changing the following keys in the scrape job can break time series continuity
- Any label set by
Evaluation of alert rules forwarded through the
prometheus_scrape interface are automatically limited to the charmed application that provided these rules. This ensures that alert rule evaluation is scoped down to the charm providing the rules.
alertmanager: interface: alertmanager_dispatch
The Alertmanager Charm aggregates, deduplicates, groups and routes alerts to selected "receivers". Alertmanager receives its alerts from Prometheus and this interaction is set up and configured using the
alertmanager relation through the
alertmanager_dispatch interface. Over this relation the Alertmanager charm keeps Prometheus informed of all Alertmanager instances (units) to which alerts must be forwarded. If your charm sets any alert rules then almost always it would need a relation with an Alertmanager charm which had been configured to forward alerts to specific receivers. In the absence of such a relation alerts even when raised will only be visible in the Prometheus user interface. A prudent approach to setting up an Observability stack is to do so in a manner such that it draws your attention to alarms as and when they are raised, without you having to periodically check a dashboard.
ingress: interface: ingress_per_unit limit: 1
Interactions with the Prometheus charm can not be assumed to originate within the same Juju model, let alone the same Kubernetes cluster, or even the same Juju cloud. Hence the Prometheus charm also supports an Ingress relation. There are multiple use cases that require an ingress, in particular
- Using the Prometheus remote write endpoint across network boundaries.
- Querying the Prometheus HTTP API endpoint across network boundaries.
- Self monitoring of Prometheus that must happen across network boundaries to ensure robustness of self monitoring.
- Supporting the Loki push API.
- Exposing the Prometheus remote write endpoint to Grafana agent.
Prometheus typical needs a "per unit" Ingress. This per unit ingress is necessary since Prometheus exposes a remote write endpoint on a per unit basis. A per unit ingress relation is available in the traefik-k8s charm and this Prometheus charm does support that relation over
catalogue: interface: catalogue
Through this relation, Prometheus provides its URL to Catalogue K8s Charmed Operator which is a landing page that helps users to locate the user interfaces of charms it relates to.
grafana-source: interface: grafana_datasource
The Grafana Charm provides a data visualization solution for metrics aggregated by Prometheus and supports the creation of bespoke dashboards for such visualization. Grafana requires a data source for its dashboards and this Prometheus charm provides the data source through the
grafana-source relation using the
grafana_datasource interface. To visualize your charms metrics using Grafana the following steps are required
- Add a relation between your charm (say
cassandra-k8s) and Prometheus so that Prometheus can aggregate the metrics.
- Add a relation between the Grafana and Prometheus charm so that metrics are forwarded to Grafana.
- Add a relation between your charm and Grafana so that your charm can forward dashboards for its metrics to Grafana.
juju relate cassandra-k8s prometheus-k8s juju relate prometheus-k8s grafana-k8s juju relate cassandra-k8s grafana-k8s
receive-remote-write: interface: prometheus_remote_write
Metrics may also be pushed to this Prometheus charm through the
receive-remote-write relation using the
prometheus_remote_write interface, which can be used with the Grafana Agent Charm to have metrics scraped by the Grafana Agent sent over to Prometheus.
Self metrics endpoint
self-metrics-endpoint: interface: prometheus_scrape
This Prometheus charm may forward information about its metrics endpoint and associated alert rules to another Prometheus charm over the
self-metrics-endpoint relation using the
prometheus_scrape interface. In order for these metrics to be aggregated by the remote Prometheus charm all that is required is to relate the two charms as in:
juju relate \ prometheus-k8s:self-metrics-endpoint \ remote-prometheus-charm:metrics-endpoint
grafana-dashboard: interface: grafana_dashboard
grafana-dashboard relation using the
grafana-dashboard interface, this Prometheus charm also provides meaningful dashboards about its metrics to be shown in a Grafana Charm .
In order to add these dashboards to Grafana all that is required is to relate the two charms in the following way:
juju relate \ prometheus-k8s:grafana-dashboard \ grafana-k8s:grafana-dashboard
This charm by default uses the latest version of the ubuntu/prometheus image.