Grafana Agent

  • By Canonical Observability
Channel Revision Published Runs on
latest/stable 75 04 Jun 2024
Ubuntu 22.04 Ubuntu 20.04
latest/stable 45 12 Dec 2023
Ubuntu 22.04 Ubuntu 20.04
latest/candidate 75 04 Jun 2024
Ubuntu 22.04 Ubuntu 20.04
latest/candidate 45 22 Nov 2023
Ubuntu 22.04 Ubuntu 20.04
latest/beta 80 28 Jun 2024
Ubuntu 22.04 Ubuntu 20.04
latest/beta 75 22 May 2024
Ubuntu 22.04 Ubuntu 20.04
latest/beta 45 22 Nov 2023
Ubuntu 22.04 Ubuntu 20.04
latest/edge 80 06 Jun 2024
Ubuntu 22.04 Ubuntu 20.04
latest/edge 75 03 May 2024
Ubuntu 22.04 Ubuntu 20.04
latest/edge 45 15 Sep 2023
Ubuntu 22.04 Ubuntu 20.04
1.0/stable 51 12 Dec 2023
Ubuntu 22.04 Ubuntu 20.04
1.0/stable 45 12 Dec 2023
Ubuntu 22.04 Ubuntu 20.04
1.0/candidate 51 22 Nov 2023
Ubuntu 22.04 Ubuntu 20.04
1.0/candidate 45 22 Nov 2023
Ubuntu 22.04 Ubuntu 20.04
1.0/beta 51 22 Nov 2023
Ubuntu 22.04 Ubuntu 20.04
1.0/beta 45 22 Nov 2023
Ubuntu 22.04 Ubuntu 20.04
1.0/edge 51 22 Nov 2023
Ubuntu 22.04 Ubuntu 20.04
1.0/edge 45 22 Nov 2023
Ubuntu 22.04 Ubuntu 20.04
juju deploy grafana-agent-k8s --channel 1.0/stable
Show information

Platform:

Sometimes applications are instrumented with more metrics than we want to afford, from a resources perspective. In such cases we can choose to selectively drop some metrics before they are ingested.

Discover high-cardinality metrics

Prometheus’s /tsdb-status endpoint is quite handy for discovering high cardinality metrics you may want to drop prior to ingestion.

In this guide, we will show how to drop all the go_gc_* metrics. To check how many go_gc_* metrics you currently have, run the following query:

count by (__name__) ({__name__=~"go_gc_.+"})

Means for dropping metrics

Metrics can be dropped by using the drop action in several different places:

  • Under <scrape_config> section (<metric_relabel_configs> subsection). For example: all the self-monitoring scrape jobs that e.g. COS Lite has in place.
  • Under <remote_write> section (<write_relabel_configs> subsection). For example: prometheus can be told to drop metrics before pushing them to another prometheus over remote-write API. This use case is not addressed in this guide.

Drop metrics via metric name in scrape config

Charms that integrate with prometheus or grafana agent, provide a “scrape config” to MetricsEndpointProvider (imported from charms.prometheus_k8s.v0.prometheus_scrape).

Let’s take for example the alertmanager self-metrics that prometheus scrapes. If we do not want prometheus or grafana agent to ingest any go_gc_* metrics from alertmanager, then we need to adjust the scrape job specified in the alertmanager charm:

diff --git a/src/charm.py b/src/charm.py
index fa3678c..f0e943b 100755
--- a/src/charm.py
+++ b/src/charm.py
@@ -250,6 +250,13 @@ class AlertmanagerCharm(CharmBase):
             "scheme": metrics_endpoint.scheme,
             "metrics_path": metrics_path,
             "static_configs": [{"targets": [target]}],
+            "metric_relabel_configs": [
+                {
+                    "source_labels": ["__name__"],
+                    "regex": "go_gc_.+",
+                    "action": "drop",
+                }
+            ]
         }
 
         return [config]

Drop metrics via the scrape-config charm

In a typical scrape-config deployment such as:

graph LR
some-external-target --- scrape-target --- scrape-config --- prometheus

We can specify the drop action via a config option for the scrape-config charm:

$ juju config sc metric_relabel_configs="$(cat <<EOF
- source_labels: ["__name__"]
  regex: "go_gc_.+"
  action: "drop"
EOF
)"

References


Help improve this document in the forum (guidelines). Last updated 2 months ago.