--- title: Charmhub | Deploy Canonical Observability Stack Lite using Charmhub - The Open Operator Collection description: Deploy the latest version of Canonical Observability Stack Lite on any cloud. url: https://charmhub.io/cos-lite/docs/tutorials/sync-alert-rules-from-git --- ##### We've discontinued the registration of new Bundles New Bundle registrations are no longer accepted. Existing bundles remain functional. We recommend using the Juju Terraform Provider for new deployments. [Learn more](https://discourse.charmhub.io/t/discontinuing-new-charmhub-bundle-registrations/15344) # Canonical Observability Stack Lite [Canonical Observability](https://charmhub.io/publisher/observability "View all packages from Canonical Observability") | bundle * [Canonical Observability](https://charmhub.io/publisher/observability "View all packages from Canonical Observability") | bundle Platform: stable 26 ``` juju deploy cos-lite ``` [Learn to deploy on juju >](https://juju.is/docs/juju/manage-applications) --- #### Relevant links * [Homepage](https://github.com/canonical/cos-lite-bundle) --- #### Contacts * [Submit a bug](https://github.com/canonical/cos-lite-bundle/issues) --- Share your thoughts on this charm with the community on discourse. [Join the discussion](https://discourse.charmhub.io/) --- [Add docs to a charm](https://juju.is/docs/sdk/add-docs-to-your-charmhub-page) Metadata | Key | Value | | --- | --- | | Summary | Sync alert rules from a git repo using the cos-configuration charm | | Categories | integrations | | Difficulty | 2 | | Author | [Jose Massón](mailto:jose.masson@canonical.com) | Imagine your business is looking to migrate your observability solution to COS Lite, but that you already have a lot of time invested into converting operational knowledge into alert rules that you’re versioning in a git repository. In this tutorial you will learn how to make COS Lite automatically sync the alert rules of your git repo to Prometheus using the [COS Configuration charm](https://charmhub.io/cos-configuration-k8s/docs). ## [Prerequisites](https://charmhub.io/cos-lite/docs/tutorials/sync-alert-rules-from-git#p-23407-prerequisites) This tutorial assumes that you already have the following: * a Kubernetes deployment [bootstrapped with Juju](https://juju.is/docs/olm/get-started-with-juju#heading--prepare-your-cloud). * a git repo with your Prometheus Alert rules. ## [Contents](https://charmhub.io/cos-lite/docs/tutorials/sync-alert-rules-from-git#p-23407-contents) 1. [Create a juju model](https://charmhub.io/cos-lite/docs/tutorials/sync-alert-rules-from-git#step-1-create-a-juju-model) 2. [Deploy Prometheus Charmed Operator](https://charmhub.io/cos-lite/docs/tutorials/sync-alert-rules-from-git#step-2-deploy-prometheus-charmed-operator) 3. [Deploy COS configuration Charmed Operator](https://charmhub.io/cos-lite/docs/tutorials/sync-alert-rules-from-git#step-3-deploy-cos-configuration-charmed-operator) 4. [Relate Prometheus to COS Configuration](https://charmhub.io/cos-lite/docs/tutorials/sync-alert-rules-from-git#step-4-relate-prometheus-to-cos-configuration) 5. [Verification](https://charmhub.io/cos-lite/docs/tutorials/sync-alert-rules-from-git#step-5-verification) 6. [Sync interval](https://charmhub.io/cos-lite/docs/tutorials/sync-alert-rules-from-git#sync-interval) 7. [Extra information](https://charmhub.io/cos-lite/docs/tutorials/sync-alert-rules-from-git#extra-information) ## [Create a juju model](https://charmhub.io/cos-lite/docs/tutorials/sync-alert-rules-from-git#p-23407-create-a-juju-model) In the following Juju model we will deploy our applications ``` juju add-model cos ``` Now, you can check that our model is empty and ready: ``` $ juju status Model Controller Cloud/Region Version SLA Timestamp cos charm-dev-batteries microk8s/localhost 3.0.3 unsupported 17:21:00-03:00 Model "admin/cos" is empty. ``` ## [Deploy Prometheus Charmed Operator](https://charmhub.io/cos-lite/docs/tutorials/sync-alert-rules-from-git#p-23407-deploy-prometheus-charmed-operator) ``` $ juju deploy prometheus-k8s prometheus --channel stable ``` After a few seconds, our Prometheus application is up and running: ``` $ juju status --relations Model Controller Cloud/Region Version SLA Timestamp cos charm-dev-batteries microk8s/localhost 3.0.3 unsupported 17:26:56-03:00 App Version Status Scale Charm Channel Rev Address Exposed Message prometheus 2.33.5 active 1 prometheus-k8s stable 103 10.152.183.235 no Unit Workload Agent Address Ports Message prometheus/0* active idle 10.1.36.122 Relation provider Requirer Interface Type Message prometheus:prometheus-peers prometheus:prometheus-peers prometheus_peers peer ``` ## [Deploy COS configuration Charmed Operator](https://charmhub.io/cos-lite/docs/tutorials/sync-alert-rules-from-git#p-23407-deploy-cos-configuration-charmed-operator) When deploying this charm, we will also need to provide some configuration: * `git_repo`: URL to repo to clone and sync against. * `git_branch`: The git branch to check out. * `git_depth`: Cloning depth, to truncate commit history to the specified number of commits. Zero means no truncating. * `prometheus_alert_rules_path`: Relative path in repo to prometheus rules. Now, let’s deploy it: ``` $ juju deploy cos-configuration-k8s cos-config \ --config git_repo=https://github.com/Abuelodelanada/cos-config \ --config git_branch=main \ --config git_depth=1 \ --config prometheus_alert_rules_path=rules/prod/prometheus/ ``` At this moment, our model looks like this: ``` $ juju status --relations Model Controller Cloud/Region Version SLA Timestamp cos charm-dev-batteries microk8s/localhost 3.0.3 unsupported 17:43:03-03:00 App Version Status Scale Charm Channel Rev Address Exposed Message cos-config 3.5.0 active 1 cos-configuration-k8s stable 15 10.152.183.147 no prometheus 2.33.5 active 1 prometheus-k8s stable 103 10.152.183.235 no Unit Workload Agent Address Ports Message cos-config/0* active idle 10.1.36.124 prometheus/0* active idle 10.1.36.122 Relation provider Requirer Interface Type Message cos-config:replicas cos-config:replicas cos_configuration_replica peer prometheus:prometheus-peers prometheus:prometheus-peers prometheus_peers peer ``` ## [Relate Prometheus to COS Configuration](https://charmhub.io/cos-lite/docs/tutorials/sync-alert-rules-from-git#p-23407-relate-prometheus-to-cos-configuration) Once we execute this command: ``` $ juju relate prometheus cos-config ``` we will get both charms related: ``` $ juju status --relations Model Controller Cloud/Region Version SLA Timestamp cos charm-dev-batteries microk8s/localhost 3.0.3 unsupported 17:51:33-03:00 App Version Status Scale Charm Channel Rev Address Exposed Message cos-config 3.5.0 active 1 cos-configuration-k8s stable 15 10.152.183.147 no prometheus 2.33.5 active 1 prometheus-k8s stable 103 10.152.183.235 no Unit Workload Agent Address Ports Message cos-config/0* active idle 10.1.36.124 prometheus/0* active idle 10.1.36.122 Relation provider Requirer Interface Type Message cos-config:prometheus-config prometheus:metrics-endpoint prometheus_scrape regular cos-config:replicas cos-config:replicas cos_configuration_replica peer prometheus:prometheus-peers prometheus:prometheus-peers prometheus_peers peer ``` ## [Verification](https://charmhub.io/cos-lite/docs/tutorials/sync-alert-rules-from-git#p-23407-verification) After setting the `git_repo` (and optionally `git_branch`), the contents should be present in the workload container, ``` $ juju ssh --container git-sync cos-config/0 ls -l /git total 4 drwxr-xr-x 3 root root 4096 Feb 23 20:40 dd2cc335a9b5734e0adbb25681074b09a4c3a111 lrwxrwxrwx 1 root root 40 Feb 23 20:40 repo -> dd2cc335a9b5734e0adbb25681074b09a4c3a111 ``` and accessible from the charm container ``` $ juju ssh cos-config/0 ls -l /var/lib/juju/storage/content-from-git/0 total 4 drwxr-xr-x 3 root root 4096 Feb 23 20:40 dd2cc335a9b5734e0adbb25681074b09a4c3a111 lrwxrwxrwx 1 root root 40 Feb 23 20:40 repo -> dd2cc335a9b5734e0adbb25681074b09a4c3a111 ``` After relating to e.g. prometheus, rules from the synced repo should appear in app data, ``` $ juju show-unit prometheus/0 \ --format json \ | jq '."prometheus/0"."relation-info"' [ { "relation-id": 0, "endpoint": "prometheus-peers", "related-endpoint": "prometheus-peers", "application-data": {}, "local-unit": { "in-scope": true, "data": { "egress-subnets": "10.152.183.235/32", "ingress-address": "10.152.183.235", "private-address": "10.152.183.235" } } }, { "relation-id": 2, "endpoint": "metrics-endpoint", "related-endpoint": "prometheus-config", "application-data": { "alert_rules": "{\"groups\": [{\"name\": \"zinc_missing_alerts\", \"rules\": [{\"alert\": \"PepeTargetMissingRemote\", \"annotations\": {\"description\": \"A Prometheus target has disappeared. An exporter might be crashed.\\n VALUE = {{ $value }}\\n LABELS = {{ $labels }}\", \"summary\": \"Prometheus target missing (instance {{ $labels.instance }})\"}, \"expr\": \"up{juju_application=\\\"PepeApp\\\"} == 0\", \"for\": \"0m\", \"labels\": {\"gitbranch\": \"main\", \"origin\": \"github\", \"severity\": \"critical\"}}]}]}" ... ``` as well as in prometheus itself using the command line: ``` $ juju ssh prometheus/0 curl localhost:9090/api/v1/rules {"status":"success","data":{"groups":[{"name":"zinc_missing_alerts","file":"/etc/prometheus/rules/juju_zinc_missing_alerts.rules","rules":[{"state":"inactive","name":"PepeTargetMissingRemote","query":"up{juju_application=\"PepeApp\"} == 0","duration":0,"labels":{"gitbranch":"main","origin":"github","severity":"critical"},"annotations":{"description":"A Prometheus target has disappeared. An exporter might be crashed.\n VALUE = {{ $value }}\n LABELS = {{ $labels }}","summary":"Prometheus target missing (instance {{ $labels.instance }})"},"alerts":[],"health":"ok","evaluationTime":0.000263312,"lastEvaluation":"2023-02-23T20:55:06.506298391Z","type":"alerting"}],"interval":60,"limit":0,"evaluationTime":0.000272579,"lastEvaluation":"2023-02-23T20:55:06.50629195Z"}]}} ``` or using Prometheus Web UI: ## [Sync interval](https://charmhub.io/cos-lite/docs/tutorials/sync-alert-rules-from-git#p-23407-sync-interval) The repo syncs on every [`update-status` event](https://juju.is/docs/sdk/update-status-event) or when the juju administrator manually runs the `sync-now` action. ``` $ juju run cos-config/0 sync-now Running operation 1 with 1 task - task 2 on unit-cos-config-0 Waiting for task 2... 18:10:28 Calling git-sync with --one-time... 18:10:29 Warning: I0223 21:10:28.544402 186 main.go:473] "level"=0 "msg"="starting up" "pid"=186 "args"=["/git-sync","--repo","https://github.com/Abuelodelanada/cos-config","--branch","main","--rev","HEAD","--depth","1","--root","/git","--dest","repo","--one-time"] git-sync-stdout: "" ``` ## [Extra information](https://charmhub.io/cos-lite/docs/tutorials/sync-alert-rules-from-git#p-23407-extra-information) COS Configuration Charmed Operator not only can forward alert rules to Prometheus, but also alert rules to Loki and Dashboards to Grafana. Paths to Loki alert rules and Grafana dashboard files, can also be set after deployment: ``` $ juju config cos-config loki_alert_rules_path=rules/prod/loki/ $ juju relate cos-config loki-k8s $ juju config cos-config grafana_dashboards_path=dashboards/prod/grafana/ $ juju relate cos-config grafana-k8s ``` ---