Loki
- By Canonical Observability
Channel | Revision | Published | Runs on |
---|---|---|---|
latest/stable | 60 | 31 Jan 2023 | |
latest/candidate | 60 | 25 Jan 2023 | |
latest/beta | 60 | 25 Jan 2023 | |
latest/edge | 89 | 18 May 2023 | |
1.0/stable | 60 | 31 Jan 2023 | |
1.0/candidate | 60 | 25 Jan 2023 | |
1.0/beta | 60 | 25 Jan 2023 | |
1.0/edge | 60 | 25 Jan 2023 |
juju deploy loki-k8s
You will need Juju 2.9 to be able to run this command. Learn how to upgrade to Juju 2.9.
Deploy Kubernetes operators easily with Juju, the Universal Operator Lifecycle Manager. Need a Kubernetes cluster? Install MicroK8s to create a full CNCF-certified Kubernetes system in under 60 seconds.
Platform:
charms.loki_k8s.v0.loki_push_api
-
- Last updated 18 May 2023
- Revision Library version 0
Overview.
This document explains how to use the two principal objects this library provides:
LokiPushApiProvider
: This object is meant to be used by any Charmed Operator that needs to
implement the provider side of the loki_push_api
relation interface. For instance, a Loki charm.
The provider side of the relation represents the server side, to which logs are being pushed.
LokiPushApiConsumer
: This object is meant to be used by any Charmed Operator that needs to
send log to Loki by implementing the consumer side of the loki_push_api
relation interface.
For instance, a Promtail or Grafana agent charm which needs to send logs to Loki.
LogProxyConsumer
: This object can be used by any Charmed Operator which needs to
send telemetry, such as logs, to Loki through a Log Proxy by implementing the consumer side of the
loki_push_api
relation interface.
Filtering logs in Loki is largely performed on the basis of labels. In the Juju ecosystem, Juju topology labels are used to uniquely identify the workload which generates telemetry like logs.
In order to be able to control the labels on the logs pushed this object adds a Pebble layer that runs Promtail in the workload container, injecting Juju topology labels into the logs on the fly.
LokiPushApiProvider Library Usage
This object may be used by any Charmed Operator which implements the loki_push_api
interface.
For instance, Loki or Grafana Agent.
For this purpose a charm needs to instantiate the LokiPushApiProvider
object with one mandatory
and three optional arguments.
charm
: A reference to the parent (Loki) charm.relation_name
: The name of the relation that the charm uses to interact with its clients, which implementLokiPushApiConsumer
orLogProxyConsumer
.If provided, this relation name must match a provided relation in metadata.yaml with the
loki_push_api
interface.The default relation name is "logging" for
LokiPushApiConsumer
and "log-proxy" forLogProxyConsumer
.For example, a provider's
metadata.yaml
file may look as follows:provides: logging: interface: loki_push_api
Subsequently, a Loki charm may instantiate the
LokiPushApiProvider
in its constructor as follows:from charms.loki_k8s.v0.loki_push_api import LokiPushApiProvider from loki_server import LokiServer ... class LokiOperatorCharm(CharmBase): ... def __init__(self, *args): super().__init__(*args) ... self._loki_ready() ... def _loki_ready(self): try: version = self._loki_server.version self.loki_provider = LokiPushApiProvider(self) logger.debug("Loki Provider is available. Loki version: %s", version) except LokiServerNotReadyError as e: self.unit.status = MaintenanceStatus(str(e)) except LokiServerError as e: self.unit.status = BlockedStatus(str(e))
port
: Loki Push Api endpoint port. Default value: 3100.rules_dir
: Directory to store alert rules. Default value: "/loki/rules".
The LokiPushApiProvider
object has several responsibilities:
Set the URL of the Loki Push API in the relation application data bag; the URL must be unique to all instances (e.g. using a load balancer).
Set the Promtail binary URL (
promtail_binary_zip_url
) so clients that useLogProxyConsumer
object could download and configure it.Process the metadata of the consumer application, provided via the "metadata" field of the consumer data bag, which are used to annotate the alert rules (see next point). An example for "metadata" is the following:
{'model': 'loki', 'model_uuid': '0b7d1071-ded2-4bf5-80a3-10a81aeb1386', 'application': 'promtail-k8s' }
Process alert rules set into the relation by the
LokiPushApiConsumer
objects, e.g.:'{ "groups": [{ "name": "loki_0b7d1071-ded2-4bf5-80a3-10a81aeb1386_promtail-k8s_alerts", "rules": [{ "alert": "HighPercentageError", "expr": "sum(rate({app=\"foo\", env=\"production\"} |= \"error\" [5m])) by (job) \n /\nsum(rate({app=\"foo\", env=\"production\"}[5m])) by (job)\n > 0.05 \n", "for": "10m", "labels": { "severity": "page", "juju_model": "loki", "juju_model_uuid": "0b7d1071-ded2-4bf5-80a3-10a81aeb1386", "juju_application": "promtail-k8s" }, "annotations": { "summary": "High request latency" } }] }] }'
Once these alert rules are sent over relation data, the LokiPushApiProvider
object
stores these files in the directory /loki/rules
inside the Loki charm container. After
storing alert rules files, the object will check alert rules by querying Loki API
endpoint: loki/api/v1/rules
.
If there are changes in the alert rules a loki_push_api_alert_rules_changed
event will
be emitted with details about the RelationEvent
which triggered it.
This events should be observed in the charm that uses LokiPushApiProvider
:
def __init__(self, *args):
super().__init__(*args)
...
self.loki_provider = LokiPushApiProvider(self)
self.framework.observe(
self.loki_provider.on.loki_push_api_alert_rules_changed,
self._loki_push_api_alert_rules_changed,
)
LokiPushApiConsumer Library Usage
This Loki charm interacts with its clients using the Loki charm library. Charms
seeking to send log to Loki, must do so using the LokiPushApiConsumer
object from
this charm library.
NOTE:
LokiPushApiConsumer
also depends on an additional charm library.Ensure sure you
charmcraft fetch-lib charms.observability_libs.v0.juju_topology
when using this library.
For the simplest use cases, using the LokiPushApiConsumer
object only requires
instantiating it, typically in the constructor of your charm (the one which
sends logs).
from charms.loki_k8s.v0.loki_push_api import LokiPushApiConsumer
class LokiClientCharm(CharmBase):
def __init__(self, *args):
super().__init__(*args)
...
self._loki_consumer = LokiPushApiConsumer(self)
The LokiPushApiConsumer
constructor requires two things:
A reference to the parent (LokiClientCharm) charm.
Optionally, the name of the relation that the Loki charm uses to interact with its clients. If provided, this relation name must match a required relation in metadata.yaml with the
loki_push_api
interface.This argument is not required if your metadata.yaml has precisely one required relation in metadata.yaml with the
loki_push_api
interface, as the lib will automatically resolve the relation name inspecting the using the meta information of the charm
Any time the relation between a Loki provider charm and a Loki consumer charm is
established, a LokiPushApiEndpointJoined
event is fired. In the consumer side
is it possible to observe this event with:
self.framework.observe(
self._loki_consumer.on.loki_push_api_endpoint_joined,
self._on_loki_push_api_endpoint_joined,
)
Any time there are departures in relations between the consumer charm and Loki
the consumer charm is informed, through a LokiPushApiEndpointDeparted
event, for instance:
self.framework.observe(
self._loki_consumer.on.loki_push_api_endpoint_departed,
self._on_loki_push_api_endpoint_departed,
)
The consumer charm can then choose to update its configuration in both situations.
Note that LokiPushApiConsumer does not add any labels automatically on its own. In
order to better integrate with the Canonical Observability Stack, you may want to configure your
software to add Juju topology labels. The
observability-libs library can be used to get topology
labels in charm code. See :func:LogProxyConsumer._scrape_configs
for an example of how
to do this with promtail.
LogProxyConsumer Library Usage
Let's say that we have a workload charm that produces logs, and we need to send those logs to a
workload implementing the loki_push_api
interface, such as Loki
or Grafana Agent
.
Adopting this object in a Charmed Operator consist of two steps:
Use the
LogProxyConsumer
class by instantiating it in the__init__
method of the charmed operator. There are two ways to get logs in to promtail. You can give it a list of files to read, or you can write to it using the syslog protocol.For example:
from charms.loki_k8s.v0.loki_push_api import LogProxyConsumer ... def __init__(self, *args): ... self._log_proxy = LogProxyConsumer( charm=self, log_files=LOG_FILES, container_name=PEER, enable_syslog=True ) self.framework.observe( self._log_proxy.on.promtail_digest_error, self._promtail_error, ) def _promtail_error(self, event): logger.error(event.message) self.unit.status = BlockedStatus(event.message)
Any time the relation between a provider charm and a LogProxy consumer charm is established, a
LogProxyEndpointJoined
event is fired. In the consumer side is it possible to observe this event with:self.framework.observe( self._log_proxy.on.log_proxy_endpoint_joined, self._on_log_proxy_endpoint_joined, )
Any time there are departures in relations between the consumer charm and the provider the consumer charm is informed, through a
LogProxyEndpointDeparted
event, for instance:self.framework.observe( self._log_proxy.on.log_proxy_endpoint_departed, self._on_log_proxy_endpoint_departed, )
The consumer charm can then choose to update its configuration in both situations.
Note that:
LOG_FILES
is alist
containing the log files we want to send toLoki
or
Grafana Agent
, for instance:LOG_FILES = [ "/var/log/apache2/access.log", "/var/log/alternatives.log", ]
container_name
is the name of the container in which the application is running. If in the Pod there is only one container, this argument can be omitted.You can configure your syslog software using
localhost
as the address and the methodLogProxyConsumer.syslog_port
to get the port, or, alternatively, if you are using rsyslog you may use the methodLogProxyConsumer.rsyslog_config()
.
Modify the
metadata.yaml
file to add:- The
log-proxy
relation in therequires
section:requires: log-proxy: interface: loki_push_api optional: true
- The
Once the library is implemented in a Charmed Operator and a relation is established with
the charm that implements the loki_push_api
interface, the library will inject a
Pebble layer that runs Promtail in the workload container to send logs.
By default, the promtail binary injected into the container will be downloaded from the internet. If, for any reason, the container has limited network access, you may allow charm administrators to provide their own promtail binary at runtime by adding the following snippet to your charm metadata:
resources:
promtail-bin:
type: file
description: Promtail binary for logging
filename: promtail-linux
Which would then allow operators to deploy the charm this way:
juju deploy \
./your_charm.charm \
--resource promtail-bin=/tmp/promtail-linux-amd64
If a different resource name is used, it can be specified with the promtail_resource_name
argument to the LogProxyConsumer
constructor.
The object can emit a PromtailDigestError
event:
- Promtail binary cannot be downloaded.
- The sha256 sum mismatch for promtail binary.
The object can raise a ContainerNotFoundError
event:
- No
container_name
parameter has been specified and the Pod has more than 1 container.
These can be monitored via the PromtailDigestError events via:
self.framework.observe(
self._loki_consumer.on.promtail_digest_error,
self._promtail_error,
)
def _promtail_error(self, event):
logger.error(msg)
self.unit.status = BlockedStatus(event.message)
)
Alerting Rules
This charm library also supports gathering alerting rules from all related Loki client
charms and enabling corresponding alerts within the Loki charm. Alert rules are
automatically gathered by LokiPushApiConsumer
object from a directory conventionally
named loki_alert_rules
.
This directory must reside at the top level in the src
folder of the
consumer charm. Each file in this directory is assumed to be a single alert rule
in YAML format. The file name must have the .rule
extension.
The format of this alert rule conforms to the
Loki docs.
An example of the contents of one such file is shown below.
alert: HighPercentageError
expr: |
sum(rate({%%juju_topology%%} |= "error" [5m])) by (job)
/
sum(rate({%%juju_topology%%}[5m])) by (job)
> 0.05
for: 10m
labels:
severity: page
annotations:
summary: High request latency
It is critical to use the %%juju_topology%%
filter in the expression for the alert
rule shown above. This filter is a stub that is automatically replaced by the
LokiPushApiConsumer
following Loki Client's Juju topology (application, model and its
UUID). Such a topology filter is essential to ensure that alert rules submitted by one
provider charm generates alerts only for that same charm.
The Loki charm may be related to multiple Loki client charms. Without this, filter rules submitted by one provider charm will also result in corresponding alerts for other provider charms. Hence, every alert rule expression must include such a topology filter stub.
Gathering alert rules and generating rule files within the Loki charm is easily done using
the alerts()
method of LokiPushApiProvider
. Alerts generated by Loki will automatically
include Juju topology labels in the alerts. These labels indicate the source of the alert.
The following labels are automatically added to every alert
juju_model
juju_model_uuid
juju_application
Whether alert rules files does not contain the keys alert
or expr
or there is no alert
rules file in alert_rules_path
a loki_push_api_alert_rules_error
event is emitted.
To handle these situations the event must be observed in the LokiClientCharm
charm.py file:
class LokiClientCharm(CharmBase):
def __init__(self, *args):
super().__init__(*args)
...
self._loki_consumer = LokiPushApiConsumer(self)
self.framework.observe(
self._loki_consumer.on.loki_push_api_alert_rules_error,
self._alert_rules_error
)
def _alert_rules_error(self, event):
self.unit.status = BlockedStatus(event.message)
Relation Data
The Loki charm uses both application and unit relation data to obtain information regarding Loki Push API and alert rules.
Units of consumer charm send their alert rules over app relation data using the alert_rules
key.
Index
class RelationNotFoundError
Description
Raised if there is no relation with the given name. None
Methods
RelationNotFoundError. __init__( self , relation_name: str )
class RelationInterfaceMismatchError
Description
Raised if the relation with the given name has a different interface. None
Methods
RelationInterfaceMismatchError. __init__( self , relation_name: str , expected_relation_interface: str , actual_relation_interface: str )
class RelationRoleMismatchError
Description
Raised if the relation with the given name has a different direction. None
Methods
RelationRoleMismatchError. __init__( self , relation_name: str , expected_relation_role: RelationRole , actual_relation_role: RelationRole )
class InvalidAlertRulePathError
Description
Raised if the alert rules folder cannot be found or is otherwise invalid. None
Methods
InvalidAlertRulePathError. __init__( self , alert_rules_absolute_path: Path , message: str )
class AlertRules
Utility class for amalgamating Loki alert rule files and injecting juju topology.
Description
An `AlertRules` object supports aggregating alert rules from files and directories in both official and single rule file formats using the `add_path()` method. All the alert rules read are annotated with Juju topology labels and amalgamated into a single data structure in the form of a Python dictionary using the `as_dict()` method. Such a dictionary can be easily dumped into JSON format and exchanged over relation data. The dictionary can also be dumped into YAML format and written directly into an alert rules file that is read by Loki. Note that multiple `AlertRules` objects must not be written into the same file, since Loki allows only a single list of alert rule groups per alert rules file. The official Loki format is a YAML file conforming to the Loki documentation (https://grafana.com/docs/loki/latest/api/#list-rule-groups). The custom single rule format is a subsection of the official YAML, having a single alert rule, effectively "one alert per file".
Methods
AlertRules. __init__( self , topology )
Build and alert rule object.
Arguments
AlertRules. add_path( self , path: str )
Add rules from a dir path.
Arguments
Description
AlertRules. as_dict( self )
Return standard alert rules file in dict representation.
Returns
class NoRelationWithInterfaceFoundError
Description
No relations with the given interface are found in the charm meta. None
Methods
NoRelationWithInterfaceFoundError. __init__( self , charm: CharmBase , relation_interface )
class MultipleRelationsWithInterfaceFoundError
Description
Multiple relations with the given interface are found in the charm meta. None
Methods
MultipleRelationsWithInterfaceFoundError. __init__( self , charm: CharmBase , relation_interface: str , relations: list )
class LokiPushApiEndpointDeparted
Description
Event emitted when Loki departed. None
class LokiPushApiEndpointJoined
Description
Event emitted when Loki joined. None
class LokiPushApiAlertRulesChanged
Description
Event emitted if there is a change in the alert rules. None
Methods
LokiPushApiAlertRulesChanged. __init__( self , handle , relation , relation_id , app , unit )
Pretend we are almost like a RelationEvent.
Description
LokiPushApiAlertRulesChanged. snapshot( self )
Description
LokiPushApiAlertRulesChanged. restore( self , snapshot: dict )
Description
class InvalidAlertRuleEvent
Event emitted when alert rule files are not parsable.
Description
Enables us to set a clear status on the provider.
Methods
InvalidAlertRuleEvent. __init__( self , handle , errors: str , valid: bool )
InvalidAlertRuleEvent. snapshot( self )
Description
InvalidAlertRuleEvent. restore( self , snapshot )
Description
class LokiPushApiEvents
Description
Event descriptor for events raised by `LokiPushApiProvider`. None
class LokiPushApiProvider
Description
A LokiPushApiProvider class. None
Methods
LokiPushApiProvider. __init__( self , charm , relation_name: str )
A Loki service provider.
Arguments
LokiPushApiProvider. update_endpoint( self , url: str , relation )
Triggers programmatically the update of endpoint in unit relation data.
Arguments
Description
LokiPushApiProvider. alerts( self )
Fetch alerts for all relations.
Returns
Description
class ConsumerBase
Description
Consumer's base class. None
Methods
ConsumerBase. __init__( self , charm: CharmBase , relation_name: str , alert_rules_path: str , recursive: bool , skip_alert_topology_labeling: bool )
ConsumerBase. loki_endpoints( self )
Fetch Loki Push API endpoints sent from LokiPushApiProvider through relation data.
Returns
class LokiPushApiConsumer
Description
Loki Consumer class. None
Methods
LokiPushApiConsumer. __init__( self , charm: CharmBase , relation_name: str , alert_rules_path: str , recursive: bool , skip_alert_topology_labeling: bool )
Construct a Loki charm client.
Arguments
Description
class ContainerNotFoundError
Description
Raised if the specified container does not exist. None
Methods
ContainerNotFoundError. __init__( self )
class MultipleContainersFoundError
Description
Raised if no container name is passed but multiple containers are present. None
Methods
MultipleContainersFoundError. __init__( self )
class PromtailDigestError
Description
Event emitted when there is an error with Promtail initialization. None
Methods
PromtailDigestError. __init__( self , handle , message )
PromtailDigestError. snapshot( self )
Description
PromtailDigestError. restore( self , snapshot )
Description
class LogProxyEndpointDeparted
Description
Event emitted when a Log Proxy has departed. None
class LogProxyEndpointJoined
Description
Event emitted when a Log Proxy joins. None
class LogProxyEvents
Description
Event descriptor for events raised by `LogProxyConsumer`. None
class LogProxyConsumer
LogProxyConsumer class.
Arguments
Description
The `LogProxyConsumer` object provides a method for attaching `promtail` to a workload in order to generate structured logging data from applications which traditionally log to syslog or do not have native Loki integration. The `LogProxyConsumer` can be instantiated as follows: self._log_proxy_consumer = LogProxyConsumer(self, log_files=["/var/log/messages"])
Methods
LogProxyConsumer. __init__( self , charm , log_files , relation_name: str , enable_syslog: bool , syslog_port: int , alert_rules_path: str , recursive: bool , container_name: str , promtail_resource_name )
LogProxyConsumer. syslog_port( self )
Gets the port on which promtail is listening for syslog.
Returns
LogProxyConsumer. rsyslog_config( self )
Generates a config line for use with rsyslog.
Returns
class CosTool
Description
Uses cos-tool to inject label matchers into alert rule expressions and validate rules. None
Methods
CosTool. __init__( self , charm )
CosTool. path( self )
Description
CosTool. apply_label_matchers( self , rules )
Description
CosTool. validate_alert_rules( self , rules: dict )
Description
CosTool. inject_label_matchers( self , expression , topology )
Description