How to enable monitoring (COS)

Both Charmed Kafka K8s and Charmed ZooKeeper K8s comes with the JMX exporter. The metrics can be queried by accessing the http://<kafka-unit-ip>:9101/metrics and http://<zookeeper-unit-ip>:9998/metrics endpoints, respectively.

Additionally, the charm provides integration with the Canonical Observability Stack.

Prerequisites

• A deployed Charmed Kafka K8s and Charmed ZooKeeper K8s bundle
• A deployed cos-lite bundle in a Kubernetes environment

Offer interfaces via the COS controller

First, we will switch to the COS K8s environment and offer COS interfaces to be cross-model integrated with the Charmed Kafka K8s model.

To switch to the Kubernetes controller for the COS model, run

juju switch <k8s_cos_controller>:<cos_model_name>

To offer the COS interfaces, run

juju offer grafana:grafana-dashboard grafana-dashboards
juju offer loki:logging loki-logging
juju offer prometheus:receive-remote-write prometheus-receive-remote-write

Consume offers via the Kafka model

Next, we will switch to the Charmed Kafka K8s model, find offers, and consume them.

We are currently on the Kubernetes controller for the COS model. To switch to the Kafka model, run

juju switch <k8s_db_controller>:<kafka_model_name>

To find offers, run the following command (make sure not to miss the “:” at the end!):

juju find-offers <k8s_cos_controller>: 

The output should be similar to the sample below, where k8s is the k8s controller name and cos is the model where cos-lite has been deployed:

Store     URL                                        Access  Interfaces
k8s  admin/cos.grafana-dashboards               admin   grafana_dashboard:grafana-dashboard
k8s  admin/cos.loki-logging                     admin   loki_push_api:logging
k8s  admin/cos.prometheus-receive-remote-write  admin   prometheus_remote_write:receive-remote-write
...

To consume offers to be reachable in the current model, run

juju consume <k8s_cos_controller>:admin/<cos_model_name>.grafana-dashboards
juju consume <k8s_cos_controller>:admin/<cos_model_name>.loki-logging
juju consume <k8s_cos_controller>:admin/<cos_model_name>.prometheus-receive-remote-write

Deploy and integrate Grafana

First, deploy grafana-agent-k8s:

juju deploy grafana-agent-k8s --trust

Then, integrate grafana-agent-k8s with consumed COS offers:

juju integrate grafana-agent-k8s grafana-dashboards
juju integrate grafana-agent-k8s loki-logging
juju integrate grafana-agent-k8s prometheus-receive-remote-write

Finally, integrate (previously known as “relate”) it with Charmed Kafka K8s

juju integrate grafana-agent-k8s kafka-k8s:grafana-dashboard
juju integrate grafana-agent-k8s kafka-k8s:logging
juju integrate grafana-agent-k8s kafka-k8s:metrics-endpoint

and Charmed ZooKeeper K8s

juju integrate grafana-agent-k8s zookeeper-k8s:grafana-dashboard
juju integrate grafana-agent-k8s zookeeper-k8s:logging
juju integrate grafana-agent-k8s zookeeper-k8s:metrics-endpoint

Wait for all components to settle down on a active/idle state on both models, e.g. <kafka_model_name> and <cos_model_name>.

After this is complete, the monitoring COS stack should be up and running and ready to be used.

Connect Grafana web interface

To connect to the Grafana web interface, follow the Browse dashboards section of the MicroK8s “Getting started” guide.

juju run grafana/leader get-admin-password --model <k8s_cos_controller>:<cos_model_name>

Tune server logging level

In order to tune the level of the server logs for Kafka and ZooKeeper, configure the log-level and log_level properties accordingly

Kafka

juju config kafka log_level=<LOG_LEVEL>

Possible values are ERROR, WARNING, INFO, DEBUG.

ZooKeeper

juju config kafka log-level=<LOG_LEVEL>

Possible values are ERROR, WARNING, INFO, DEBUG.