Sync alert rules from a git repo using the cos-configuration charm
Metadata
| Key | Value |
|---|---|
| Summary | Sync alert rules from a git repo using the cos-configuration charm |
| Categories | integrations |
| Difficulty | 2 |
| Author | Jose Massón |
Imagine your business is looking to migrate your observability solution to COS Lite, but that you already have a lot of time invested into converting operational knowledge into alert rules that you’re versioning in a git repository.
In this tutorial you will learn how to make COS Lite automatically sync the alert rules of your git repo to Prometheus using the COS Configuration charm.
Prerequisites
This tutorial assumes that you already have the following:
- a Kubernetes deployment bootstrapped with Juju.
- a git repo with your Prometheus Alert rules.
Contents
- Create a juju model
- Deploy Prometheus Charmed Operator
- Deploy COS configuration Charmed Operator
- Relate Prometheus to COS Configuration
- Verification
- Sync interval
- Extra information
Create a juju model
In the following Juju model we will deploy our applications
juju add-model cos
Now, you can check that our model is empty and ready:
$ juju status
Model Controller Cloud/Region Version SLA Timestamp
cos charm-dev-batteries microk8s/localhost 3.0.3 unsupported 17:21:00-03:00
Model "admin/cos" is empty.
Deploy Prometheus Charmed Operator
$ juju deploy prometheus-k8s prometheus --channel stable
After a few seconds, our Prometheus application is up and running:
$ juju status --relations
Model Controller Cloud/Region Version SLA Timestamp
cos charm-dev-batteries microk8s/localhost 3.0.3 unsupported 17:26:56-03:00
App Version Status Scale Charm Channel Rev Address Exposed Message
prometheus 2.33.5 active 1 prometheus-k8s stable 103 10.152.183.235 no
Unit Workload Agent Address Ports Message
prometheus/0* active idle 10.1.36.122
Relation provider Requirer Interface Type Message
prometheus:prometheus-peers prometheus:prometheus-peers prometheus_peers peer
Deploy COS configuration Charmed Operator
When deploying this charm, we will also need to provide some configuration:
git_repo: URL to repo to clone and sync against.git_branch: The git branch to check out.git_depth: Cloning depth, to truncate commit history to the specified number of commits. Zero means no truncating.prometheus_alert_rules_path: Relative path in repo to prometheus rules.
Now, let’s deploy it:
$ juju deploy cos-configuration-k8s cos-config \
--config git_repo=https://github.com/Abuelodelanada/cos-config \
--config git_branch=main \
--config git_depth=1 \
--config prometheus_alert_rules_path=rules/prod/prometheus/
At this moment, our model looks like this:
$ juju status --relations
Model Controller Cloud/Region Version SLA Timestamp
cos charm-dev-batteries microk8s/localhost 3.0.3 unsupported 17:43:03-03:00
App Version Status Scale Charm Channel Rev Address Exposed Message
cos-config 3.5.0 active 1 cos-configuration-k8s stable 15 10.152.183.147 no
prometheus 2.33.5 active 1 prometheus-k8s stable 103 10.152.183.235 no
Unit Workload Agent Address Ports Message
cos-config/0* active idle 10.1.36.124
prometheus/0* active idle 10.1.36.122
Relation provider Requirer Interface Type Message
cos-config:replicas cos-config:replicas cos_configuration_replica peer
prometheus:prometheus-peers prometheus:prometheus-peers prometheus_peers peer
Relate Prometheus to COS Configuration
Once we execute this command:
$ juju relate prometheus cos-config
we will get both charms related:
$ juju status --relations
Model Controller Cloud/Region Version SLA Timestamp
cos charm-dev-batteries microk8s/localhost 3.0.3 unsupported 17:51:33-03:00
App Version Status Scale Charm Channel Rev Address Exposed Message
cos-config 3.5.0 active 1 cos-configuration-k8s stable 15 10.152.183.147 no
prometheus 2.33.5 active 1 prometheus-k8s stable 103 10.152.183.235 no
Unit Workload Agent Address Ports Message
cos-config/0* active idle 10.1.36.124
prometheus/0* active idle 10.1.36.122
Relation provider Requirer Interface Type Message
cos-config:prometheus-config prometheus:metrics-endpoint prometheus_scrape regular
cos-config:replicas cos-config:replicas cos_configuration_replica peer
prometheus:prometheus-peers prometheus:prometheus-peers prometheus_peers peer
Verification
After setting the git_repo (and optionally git_branch), the contents should be present in the workload container,
$ juju ssh --container git-sync cos-config/0 ls -l /git
total 4
drwxr-xr-x 3 root root 4096 Feb 23 20:40 dd2cc335a9b5734e0adbb25681074b09a4c3a111
lrwxrwxrwx 1 root root 40 Feb 23 20:40 repo -> dd2cc335a9b5734e0adbb25681074b09a4c3a111
and accessible from the charm container
$ juju ssh cos-config/0 ls -l /var/lib/juju/storage/content-from-git/0
total 4
drwxr-xr-x 3 root root 4096 Feb 23 20:40 dd2cc335a9b5734e0adbb25681074b09a4c3a111
lrwxrwxrwx 1 root root 40 Feb 23 20:40 repo -> dd2cc335a9b5734e0adbb25681074b09a4c3a111
After relating to e.g. prometheus, rules from the synced repo should appear in app data,
$ juju show-unit prometheus/0 \
--format json \
| jq '."prometheus/0"."relation-info"'
[
{
"relation-id": 0,
"endpoint": "prometheus-peers",
"related-endpoint": "prometheus-peers",
"application-data": {},
"local-unit": {
"in-scope": true,
"data": {
"egress-subnets": "10.152.183.235/32",
"ingress-address": "10.152.183.235",
"private-address": "10.152.183.235"
}
}
},
{
"relation-id": 2,
"endpoint": "metrics-endpoint",
"related-endpoint": "prometheus-config",
"application-data": {
"alert_rules": "{\"groups\": [{\"name\": \"zinc_missing_alerts\", \"rules\": [{\"alert\": \"PepeTargetMissingRemote\", \"annotations\": {\"description\": \"A Prometheus target has disappeared. An exporter might be crashed.\\n VALUE = {{ $value }}\\n LABELS = {{ $labels }}\", \"summary\": \"Prometheus target missing (instance {{ $labels.instance }})\"}, \"expr\": \"up{juju_application=\\\"PepeApp\\\"} == 0\", \"for\": \"0m\", \"labels\": {\"gitbranch\": \"main\", \"origin\": \"github\", \"severity\": \"critical\"}}]}]}"
...
as well as in prometheus itself using the command line:
$ juju ssh prometheus/0 curl localhost:9090/api/v1/rules
{"status":"success","data":{"groups":[{"name":"zinc_missing_alerts","file":"/etc/prometheus/rules/juju_zinc_missing_alerts.rules","rules":[{"state":"inactive","name":"PepeTargetMissingRemote","query":"up{juju_application=\"PepeApp\"} == 0","duration":0,"labels":{"gitbranch":"main","origin":"github","severity":"critical"},"annotations":{"description":"A Prometheus target has disappeared. An exporter might be crashed.\n VALUE = {{ $value }}\n LABELS = {{ $labels }}","summary":"Prometheus target missing (instance {{ $labels.instance }})"},"alerts":[],"health":"ok","evaluationTime":0.000263312,"lastEvaluation":"2023-02-23T20:55:06.506298391Z","type":"alerting"}],"interval":60,"limit":0,"evaluationTime":0.000272579,"lastEvaluation":"2023-02-23T20:55:06.50629195Z"}]}}
or using Prometheus Web UI:
Sync interval
The repo syncs on every update-status event or when the juju administrator manually runs the sync-now action.
$ juju run cos-config/0 sync-now
Running operation 1 with 1 task
- task 2 on unit-cos-config-0
Waiting for task 2...
18:10:28 Calling git-sync with --one-time...
18:10:29 Warning: I0223 21:10:28.544402 186 main.go:473] "level"=0 "msg"="starting up" "pid"=186 "args"=["/git-sync","--repo","https://github.com/Abuelodelanada/cos-config","--branch","main","--rev","HEAD","--depth","1","--root","/git","--dest","repo","--one-time"]
git-sync-stdout: ""
Extra information
COS Configuration Charmed Operator not only can forward alert rules to Prometheus, but also alert rules to Loki and Dashboards to Grafana.
Paths to Loki alert rules and Grafana dashboard files, can also be set after deployment:
$ juju config cos-config loki_alert_rules_path=rules/prod/loki/
$ juju relate cos-config loki-k8s
$ juju config cos-config grafana_dashboards_path=dashboards/prod/grafana/
$ juju relate cos-config grafana-k8s
Last updated 4 months ago.