Grafana Agent
- By Canonical Observability
Channel | Revision | Published | Runs on |
---|---|---|---|
latest/stable | 75 | 04 Jun 2024 | |
latest/stable | 45 | 12 Dec 2023 | |
latest/candidate | 80 | 01 Aug 2024 | |
latest/candidate | 75 | 04 Jun 2024 | |
latest/candidate | 45 | 22 Nov 2023 | |
latest/beta | 75 | 22 May 2024 | |
latest/beta | 45 | 22 Nov 2023 | |
latest/edge | 89 | 27 Aug 2024 | |
latest/edge | 75 | 03 May 2024 | |
latest/edge | 45 | 15 Sep 2023 | |
1.0/stable | 51 | 12 Dec 2023 | |
1.0/stable | 45 | 12 Dec 2023 | |
1.0/candidate | 51 | 22 Nov 2023 | |
1.0/candidate | 45 | 22 Nov 2023 | |
1.0/beta | 51 | 22 Nov 2023 | |
1.0/beta | 45 | 22 Nov 2023 | |
1.0/edge | 51 | 22 Nov 2023 | |
1.0/edge | 45 | 22 Nov 2023 |
juju deploy grafana-agent-k8s --channel 1.0/edge
Deploy Kubernetes operators easily with Juju, the Universal Operator Lifecycle Manager. Need a Kubernetes cluster? Install MicroK8s to create a full CNCF-certified Kubernetes system in under 60 seconds.
Platform:
Sometimes applications are instrumented with more metrics than we want to afford, from a resources perspective. In such cases we can choose to selectively drop some metrics before they are ingested.
Discover high-cardinality metrics
Prometheus’s /tsdb-status
endpoint is quite handy for discovering high cardinality metrics you may want to drop prior to ingestion.
In this guide, we will show how to drop all the go_gc_*
metrics. To check how many go_gc_*
metrics you currently have, run the following query:
count by (__name__) ({__name__=~"go_gc_.+"})
Means for dropping metrics
Metrics can be dropped by using the drop
action in several different places:
- Under
<scrape_config>
section (<metric_relabel_configs>
subsection). For example: all the self-monitoring scrape jobs that e.g. COS Lite has in place. - Under
<remote_write>
section (<write_relabel_configs>
subsection). For example: prometheus can be told to drop metrics before pushing them to another prometheus over remote-write API. This use case is not addressed in this guide.
Drop metrics via metric name in scrape config
Charms that integrate with prometheus or grafana agent, provide a “scrape config” to MetricsEndpointProvider
(imported from charms.prometheus_k8s.v0.prometheus_scrape
).
Let’s take for example the alertmanager self-metrics that prometheus scrapes. If we do not want prometheus or grafana agent to ingest any go_gc_*
metrics from alertmanager, then we need to adjust the scrape job specified in the alertmanager charm:
diff --git a/src/charm.py b/src/charm.py
index fa3678c..f0e943b 100755
--- a/src/charm.py
+++ b/src/charm.py
@@ -250,6 +250,13 @@ class AlertmanagerCharm(CharmBase):
"scheme": metrics_endpoint.scheme,
"metrics_path": metrics_path,
"static_configs": [{"targets": [target]}],
+ "metric_relabel_configs": [
+ {
+ "source_labels": ["__name__"],
+ "regex": "go_gc_.+",
+ "action": "drop",
+ }
+ ]
}
return [config]
Drop metrics via the scrape-config charm
In a typical scrape-config deployment such as:
graph LR
some-external-target --- scrape-target --- scrape-config --- prometheus
We can specify the drop
action via a config option for the scrape-config charm:
$ juju config sc metric_relabel_configs="$(cat <<EOF
- source_labels: ["__name__"]
regex: "go_gc_.+"
action: "drop"
EOF
)"
References
- Official docs:
<relabel_config>
- Dropping metrics at scrape time with Prometheus (robustperception, 2015)
- How relabeling in Prometheus works (grafana.com, 2022)
- How to drop and delete metrics in Prometheus (gh:tanmay-bhat, 2022)
- Playgrounds: