Canonical Observability Stack Proxy

Basic deployment

Deploy Canonical Observability Stack (Lite) on a model backed by Kubernetes, then deploy the cos-proxy operator in a reactive/machine model from which you would like to collect observability data.

The cos-proxy operator is designed to work with cross-model relations, so the first step is to establish those:

juju offer cos.prometheus:metrics-endpoint
juju offer cos.grafana:grafana-dashboard

juju consume cos.prometheus
juju consume cos.grafana

Next, deploy the cos-proxy operator in a reactive model:

juju deploy -m reactive cos-proxy-operator --channel=beta

For any charm which may export:

• Grafana dashboards
• Prometheus metrics endpoints
• Prometheus alert rules
• NRPE jobs

You may relate them to cos-proxy. “Topology” for model-driven observability will be automatically injected into Prometheus labels and Alert rule expressions.

NRPE jobs are checked via nrpe_exporter, which provides a Prometheus-compatible HTTP endpoint which knows how to speak to NRPE.

For example:

❯ juju status -m dash --color --relations
Model  Controller  Cloud/Region       Version  SLA          Timestamp
reactive   overlord    lxdremote/default  2.9.22   unsupported  16:43:14Z

SAAS        Status  Store     URL
grafana     active  overlord  admin/cos.grafana
prometheus  active  overlord  admin/cos.prometheus

App        Version  Status  Scale  Charm      Store       Channel  Rev  OS      Message
cos-proxy           active      1  cos-proxy  local                 14  ubuntu  
memcached           active      1  memcached  charmstore  stable    34  ubuntu  Unit is ready
nagios              active      1  nagios     charmstore  stable    46  ubuntu  ready
nrpe                active      2  nrpe       charmstore  stable    75  ubuntu  Ready (source version/commit cs-nrpe-...)
telegraf            active      1  telegraf   charmstore  stable    44  ubuntu  Monitoring ubuntu/0 (source version/commit 26e531a)
ubuntu     20.04    active      1  ubuntu     charmstore  stable    18  ubuntu  

Unit           Workload  Agent  Machine  Public address  Ports          Message
cos-proxy/14*  active    idle   17       10.159.132.117                 
memcached/0*   active    idle   2        10.159.132.52   11211/tcp      Unit is ready
  nrpe/1       active    idle            10.159.132.52   icmp,5666/tcp  Ready (source version/commit cs-nrpe-...)
nagios/0*      active    idle   1        10.159.132.207  80/tcp         ready
ubuntu/0*      active    idle   0        10.159.132.173                 
  nrpe/0*      active    idle            10.159.132.173  icmp,5666/tcp  Ready (source version/commit cs-nrpe-...)
  telegraf/6*  active    idle            10.159.132.173  9103/tcp       Monitoring ubuntu/0 (source version/commit 26e531a)

Machine  State    DNS             Inst id         Series  AZ  Message
0        started  10.159.132.173  juju-dda879-0   focal       Running
1        started  10.159.132.207  juju-dda879-1   bionic      Running
2        started  10.159.132.52   juju-dda879-2   focal       Running
17       started  10.159.132.117  juju-dda879-17  focal       Running

Relation provider                       Requirer                     Interface              Type         Message
cos-proxy:downstream-prometheus-scrape  prometheus:metrics-endpoint  prometheus_scrape      regular      
memcached:cluster                       memcached:cluster            memcached-replication  peer         
memcached:juju-info                     nrpe:general-info            juju-info              subordinate  
memcached:local-monitors                nrpe:local-monitors          local-monitors         subordinate  
memcached:monitors                      cos-proxy:monitors           monitors               regular      
memcached:nrpe-external-master          nrpe:nrpe-external-master    nrpe-external-master   subordinate  
nrpe:monitors                           cos-proxy:monitors           monitors               regular      
nrpe:monitors                           nagios:monitors              monitors               regular      
ubuntu:juju-info                        nrpe:general-info            juju-info              subordinate  
ubuntu:juju-info                        telegraf:juju-info           juju-info              subordinate  

Note: “system”-level NRPE jobs, such as check_conntrack are exposed on the monitors interface, while “charm”-level jobs specific to the workload are exposed on the local-monitors interface, and may not be provisioned until a nrpe-external-master relation is established.

To accurately pull all NRPE jobs from a charm, all of these relations should be established. This is an identical user experience to the nrpe charm, but it’s documented here for completeness.