Prometheus

  • By Balbir Thomas
Channel Version Revision Published Runs on
latest/candidate 55 55 08 Aug 2022
Ubuntu 20.04
latest/beta 55 55 08 Aug 2022
Ubuntu 20.04
latest/edge 70 70 22 Sep 2022
Ubuntu 20.04
juju deploy prometheus-k8s --channel candidate
Show information

Platform:

charms.prometheus_k8s.v0.prometheus_remote_write

Prometheus remote-write library.

This library facilitates the integration of the prometheus_remote_write interface.

Charms that need to push data to a charm exposing the Prometheus remote_write API, should use the PrometheusRemoteWriteConsumer. Charms that operate software that exposes the Prometheus remote_write API, that is, they can receive metrics data over remote_write, should use the PrometheusRemoteWriteProducer.


Index

class RelationNotFoundError

Description

Raised if there is no relation with the given name. None

Methods

RelationNotFoundError. __init__( self , relation_name: str )

class RelationInterfaceMismatchError

Description

Raised if the relation with the given name has a different interface. None

Methods

RelationInterfaceMismatchError. __init__( self , relation_name: str , expected_relation_interface: str , actual_relation_interface: str )

class RelationRoleMismatchError

Description

Raised if the relation with the given name has a different direction. None

Methods

RelationRoleMismatchError. __init__( self , relation_name: str , expected_relation_role: RelationRole , actual_relation_role: RelationRole )

class InvalidAlertRuleEvent

Event emitted when alert rule files are not parsable.

Description

Enables us to set a clear status on the provider.

Methods

InvalidAlertRuleEvent. __init__( self , handle , errors: str , valid: bool )

InvalidAlertRuleEvent. snapshot( self )

Description

Save alert rule information. None

InvalidAlertRuleEvent. restore( self , snapshot )

Description

Restore alert rule information. None

def _is_official_alert_rule_format(rules_dict: dict)

Are alert rules in the upstream format as supported by Prometheus.

Arguments

rules_dict
a set of alert rules in Python dictionary format

Returns

True if alert rules are in official Prometheus file format.

Description

Alert rules in dictionary format are in "official" form if they contain a "groups" key, since this implies they contain a list of alert rule groups.

def _is_single_alert_rule_format(rules_dict: dict)

Are alert rules in single rule format.

Returns

True if alert rule is in single rule file format.

Description

The Prometheus charm library supports reading of alert rules in a custom format that consists of a single alert rule per file. This does not conform to the official Prometheus alert rule file format which requires that each alert rules file consists of a list of alert rule groups and each group consists of a list of alert rules. Alert rules in dictionary form are considered to be in single rule format if in the least it contains two keys corresponding to the alert rule name and alert expression.

class AlertRules

Utility class for amalgamating prometheus alert rule files and injecting juju topology.

Description

An `AlertRules` object supports aggregating alert rules from files and directories in both official and single rule file formats using the `add_path()` method. All the alert rules read are annotated with Juju topology labels and amalgamated into a single data structure in the form of a Python dictionary using the `as_dict()` method. Such a dictionary can be easily dumped into JSON format and exchanged over relation data. The dictionary can also be dumped into YAML format and written directly into an alert rules file that is read by Prometheus. Note that multiple `AlertRules` objects must not be written into the same file, since Prometheus allows only a single list of alert rule groups per alert rules file. The official Prometheus format is a YAML file conforming to the Prometheus documentation (https://prometheus.io/docs/prometheus/latest/configuration/alerting_rules/). The custom single rule format is a subsection of the official YAML, having a single alert rule, effectively "one alert per file".

Methods

AlertRules. __init__( self , topology )

Build and alert rule object.

Arguments

topology
an optional `JujuTopology` instance that is used to annotate all alert rules.

AlertRules. _from_file( self , root_path: Path , file_path: Path )

Read a rules file from path, injecting juju topology.

Arguments

root_path
full path to the root rules folder (used only for generating group name)
file_path
full path to a *.rule file.

Returns

A list of dictionaries representing the rules file, if file is valid (the structure is formed by `yaml.safe_load` of the file); an empty list otherwise.

AlertRules. _group_name( self , root_path: str , file_path: str , group_name: str )

Generate group name from path and topology.

Arguments

root_path
path to the root rules dir.
file_path
path to rule file.
group_name
original group name to keep as part of the new augmented group name

Returns

New group name, augmented by juju topology and relative path.

Description

The group name is made up of the relative path between the root dir_path, the file path, and topology identifier.

AlertRules. _is_already_modified( self , name: str )

Description

Detect whether a group name has already been modified with juju topology. None

AlertRules. _multi_suffix_glob( cls , dir_path: Path , suffixes , recursive: bool )

Helper function for getting all files in a directory that have a matching suffix.

Arguments

dir_path
path to the directory to glob from.
suffixes
list of suffixes to include in the glob (items should begin with a period).
recursive
a flag indicating whether a glob is recursive (nested) or not.

Returns

List of files in `dir_path` that have one of the suffixes specified in `suffixes`.

AlertRules. _from_dir( self , dir_path: Path , recursive: bool )

Read all rule files in a directory.

Arguments

dir_path
directory containing *.rule files (alert rules without groups).
recursive
flag indicating whether to scan for rule files recursively.

Returns

a list of dictionaries representing prometheus alert rule groups, each dictionary representing an alert group (structure determined by `yaml.safe_load`).

Description

All rules from files for the same directory are loaded into a single group. The generated name of this group includes juju topology. By default, only the top directory is scanned; for nested scanning, pass `recursive=True`.

AlertRules. add_path( self , path: str )

Add rules from a dir path.

Arguments

path
either a rules file or a dir of rules files.
recursive
whether to read files recursively or not (no impact if `path` is a file).

Returns

True if path was added else False.

Description

All rules from files are aggregated into a data structure representing a single rule file. All group names are augmented with juju topology.

AlertRules. as_dict( self )

Return standard alert rules file in dict representation.

Returns

a dictionary containing a single list of alert rule groups. The list of alert rule groups is provided as value of the "groups" dictionary key.

def _validate_relation_by_interface_and_direction(
    charm: CharmBase,
    relation_name: str,
    expected_relation_interface: str,
    expected_relation_role: RelationRole
)

Verifies that a relation has the necessary characteristics.

Arguments

charm
a `CharmBase` object to scan for the matching relation.
relation_name
the name of the relation to be verified.
expected_relation_interface
the interface name to be matched by the relation named `relation_name`.
expected_relation_role
whether the `relation_name` must be either provided or required by `charm`.

Description

Verifies that the `relation_name` provided: (1) exists in metadata.yaml, (2) declares as interface the interface name passed as `relation_interface` and (3) has the right "direction", i.e., it is a relation that `charm` provides or requires.

class PrometheusRemoteWriteEndpointsChangedEvent

Description

Event emitted when Prometheus remote_write endpoints change. None

Methods

PrometheusRemoteWriteEndpointsChangedEvent. __init__( self , handle , relation_id )

PrometheusRemoteWriteEndpointsChangedEvent. snapshot( self )

Description

Save scrape Prometheus remote_write information. None

PrometheusRemoteWriteEndpointsChangedEvent. restore( self , snapshot )

Description

Restore scrape Prometheus remote_write information. None

class InvalidAlertRulePathError

Description

Raised if the alert rules folder cannot be found or is otherwise invalid. None

Methods

InvalidAlertRulePathError. __init__( self , alert_rules_absolute_path: str , message: str )

def _resolve_dir_against_charm_path(charm: CharmBase)

Resolve the provided path items against the directory of the main file.

Description

Look up the directory of the main .py file being executed. This is normally going to be the charm.py file of the charm including this library. Then, resolve the provided path elements and, if the result path exists and is a directory, return its absolute path; otherwise, return `None`.

class PrometheusRemoteWriteConsumerEvents

Description

Event descriptor for events raised by `PrometheusRemoteWriteConsumer`. None

class PrometheusRemoteWriteConsumer

API that manages a required `prometheus_remote_write` relation.

Description

The `PrometheusRemoteWriteConsumer` is intended to be used by charms that need to push data to other charms over the Prometheus remote_write API. The `PrometheusRemoteWriteConsumer` object can be instantiated as follows in your charm: ``` from charms.prometheus_k8s.v0.prometheus_remote_write import PrometheusRemoteWriteConsumer def __init__(self, *args): ... self.remote_write_consumer = PrometheusRemoteWriteConsumer(self) ... ``` The `PrometheusRemoteWriteConsumer` assumes that, in the `metadata.yaml` of your charm, you declare a required relation as follows: ``` requires: receive-remote-write: # Relation name interface: prometheus_remote_write # Relation interface ``` The charmed operator is expected to use the `PrometheusRemoteWriteConsumer` as follows: ``` def __init__(self, *args): ... self.remote_write_consumer = PrometheusRemoteWriteConsumer(self) ... self.framework.observe( self.remote_write_consumer.on.endpoints_changed, self._handle_endpoints_changed, ) ``` The `endpoints_changed` event will fire in situations such as provider ip change (e.g. relation created, provider upgrade, provider pod churn) or provider config change (e.g. metadata settings). Then, inside the logic of `_handle_endpoints_changed`, the updated endpoint list is retrieved with with: ``` self.remote_write_consumer.endpoints ``` which returns a dictionary structured like the Prometheus configuration object (see https://prometheus.io/docs/prometheus/latest/configuration/configuration/#remote_write). Regarding the default relation name, `receive-remote-write`: if you choose to change it, you would need to explicitly provide it to the `PrometheusRemoteWriteConsumer` via the `relation_name` constructor argument. (The relation interface, on the other hand, is fixed and, if you were to change it, your charm would not be able to relate with other charms using the correct relation interface. The library prevents you from doing that by raising an exception.) In any case, it is strongly discouraged to change the relation name: having consistent relation names across charms that do similar things is good practice and more straightforward for the users of your charm. The one exception to the rule above, is if your charm needs to both consume and provide a relation using the `prometheus_remote_write` interface, in which case changing the relation name to differentiate between "incoming" and "outgoing" remote write interactions is necessary. It is also possible to specify alert rules. By default, this library will search `<charm_parent_dir>/prometheus_alert_rules`, which in standard charm layouts resolves to `src/prometheus_alert_rules`. Each set of alert rules, grouped by the topology identifier, goes into a separate `*.rule` file. If the syntax of a rule is invalid, the `MetricsEndpointProvider` logs an error and does not load the particular rule. To avoid false positives and negatives in the evaluation of your alert rules, you must always add the `%%juju_topology%%` token as label filters in the PromQL expression, e.g.: alert: UnitUnavailable expr: up{%%juju_topology%%} < 1 for: 0m labels: severity: critical annotations: summary: Unit {{ $labels.juju_model }}/{{ $labels.juju_unit }} unavailable description: > The unit {{ $labels.juju_model }} {{ $labels.juju_unit }} is unavailable The `%%juju_topology%%` token will be replaced with label filters ensuring that the only timeseries evaluated are those scraped from this charm, and no other. Failing to ensure that the `%%juju_topology%%` token is applied to each and every of the queries timeseries will lead to unpredictable alert rule evaluation if your charm is deployed multiple times and various of its instances are monitored by the same Prometheus.

Methods

PrometheusRemoteWriteConsumer. __init__( self , charm: CharmBase , relation_name: str , alert_rules_path: str )

API to manage a required relation with the `prometheus_remote_write` interface.

Arguments

charm
The charm object that instantiated this class.
relation_name
Name of the relation with the `prometheus_remote_write` interface as defined in metadata.yaml.

PrometheusRemoteWriteConsumer. _handle_endpoints_changed( self , event: RelationEvent )

PrometheusRemoteWriteConsumer. _push_alerts_on_relation_joined( self , event: RelationEvent )

PrometheusRemoteWriteConsumer. _push_alerts_to_all_relation_databags( self , _ )

PrometheusRemoteWriteConsumer. _push_alerts_to_relation_databag( self , relation: Relation )

PrometheusRemoteWriteConsumer. reload_alerts( self )

Description

Reload alert rules from disk and push to relation data. None

PrometheusRemoteWriteConsumer. endpoints( self )

A config object ready to be dropped into a prometheus config file.

Returns

A list of dictionaries where each dictionary provides information about a single remote_write endpoint.

Description

The format of the dict is specified in the official prometheus docs: https://prometheus.io/docs/prometheus/latest/configuration/configuration/#remote_write

class PrometheusRemoteWriteProvider

API that manages a provided `prometheus_remote_write` relation.

Description

The `PrometheusRemoteWriteProvider` is intended to be used by charms whose workloads need to receive data from other charms' workloads over the Prometheus remote_write API. The `PrometheusRemoteWriteProvider` object can be instantiated as follows in your charm: ``` from charms.prometheus_k8s.v0.prometheus_remote_write import PrometheusRemoteWriteProvider def __init__(self, *args): ... self.remote_write_provider = PrometheusRemoteWriteProvider(self) ... ``` The `PrometheusRemoteWriteProvider` assumes that, in the `metadata.yaml` of your charm, you declare a provided relation as follows: ``` provides: receive-remote-write: # Relation name interface: prometheus_remote_write # Relation interface ``` About the name of the relation managed by this library: technically, you *could* change the relation name, `receive-remote-write`, but that requires you to provide the new relation name to the `PrometheusRemoteWriteProducer` via the `relation_name` constructor argument. (The relation interface, on the other hand, is immutable and, if you were to change it, your charm would not be able to relate with other charms using the right relation interface. The library prevents you from doing that by raising an exception.) In any case, it is strongly discouraged to change the relation name: having consistent relation names across charms that do similar things is a very good thing for the people that will use your charm. The one exception to the rule above, is if you charm needs to both consume and provide a relation using the `prometheus_remote_write` interface, in which case changing the relation name to differentiate between "incoming" and "outgoing" remote write interactions is necessary.

Methods

PrometheusRemoteWriteProvider. __init__( self , charm: CharmBase , relation_name: str , endpoint_schema: str , endpoint_address: str , endpoint_port , endpoint_path: str )

API to manage a provided relation with the `prometheus_remote_write` interface.

Arguments

charm
The charm object that instantiated this class.
relation_name
Name of the relation with the `prometheus_remote_write` interface as defined in metadata.yaml.
endpoint_schema
The URL schema for your remote_write endpoint. Defaults to `http`.
endpoint_address
The URL host for your remote_write endpoint as reachable from the client. This might be either the pod IP, or you might want to expose an address routable from outside the Kubernetes cluster, e.g., the host address of an Ingress. If not provided, it defaults to the unit's FQDN.
endpoint_port
The URL port for your remote_write endpoint. Defaults to `9090`.
endpoint_path
The URL path for your remote_write endpoint. Defaults to `/api/v1/write`.

PrometheusRemoteWriteProvider. _on_relation_change( self , event: RelationEvent )

PrometheusRemoteWriteProvider. update_endpoint( self , relation: Relation )

Triggers programmatically the update of the relation data.

Arguments

relation
An optional instance of `class:ops.model.Relation` to update. If not provided, all instances of the `prometheus_remote_write` relation are updated.

Description

This method should be used when the charm relying on this library needs to update the relation data in response to something occurring outside of the `prometheus_remote_write` relation lifecycle, e.g., in case of a host address change because the charmed operator becomes connected to an Ingress after the `prometheus_remote_write` relation is established.

PrometheusRemoteWriteProvider. _set_endpoint_on_relation( self , relation: Relation )

Set the remote_write endpoint on relations.

Arguments

relation
The relation whose data to update.

PrometheusRemoteWriteProvider. alerts( self )

Fetch alert rules from all relations.

Returns

a dictionary mapping the name of an alert rule group to the group.

Description

A Prometheus alert rules file consists of a list of "groups". Each group consists of a list of alerts (`rules`) that are sequentially executed. This method returns all the alert rules provided by each related metrics provider charm. These rules may be used to generate a separate alert rules file for each relation since the returned list of alert groups are indexed by relation ID. Also for each relation ID associated scrape metadata such as Juju model, UUID and application name are provided so the a unique name may be generated for the rules file. For each relation the structure of data returned is a dictionary with four keys - groups - model - model_uuid - application The value of the `groups` key is such that it may be used to generate a Prometheus alert rules file directly using `yaml.dump` but the `groups` key itself must be included as this is required by Prometheus, for example as in `yaml.dump({"groups": alerts["groups"]})`. The `PrometheusRemoteWriteProvider` accepts a list of rules and these rules are all placed into one group.

class CosTool

Description

Uses cos-tool to inject label matchers into alert rule expressions and validate rules. None

Methods

CosTool. __init__( self , charm )

CosTool. path( self )

Description

Lazy lookup of the path of cos-tool. None

CosTool. apply_label_matchers( self , rules )

Description

Will apply label matchers to the expression of all alerts in all supplied groups. None

CosTool. validate_alert_rules( self , rules: dict )

Description

Will validate correctness alert rules, returning a boolean and any errors. None

CosTool. inject_label_matchers( self , expression , topology )

Description

Add label matchers to an expression. None

CosTool. _get_tool_path( self )

CosTool. _exec( self , cmd )