Monitoring (COS)

The Canonical Observability Stack (COS) is a set of tools that facilitates gathering, processing, visualizing, and setting up alerts on telemetry signals generated by workloads in and outside of Juju.

The OpenSearch charm can use COS to connect to Grafana and Prometheus to use monitoring, alert rules, and log features.

See: How to enable monitoring via COS and Grafana

Summary

• Metrics
• Alert rules
• Logs

Metrics

Prometheus metrics are automatically installed as an OpenSearch plugin: The Prometheus Exporter Plugin for OpenSearch

The meaning of the metrics collected can be found in the upstream documentation:

• indices stats metrics
• nodes_stats_metrics
• cluster_stats_metrics

Alert Rules

The charm deploys a pre-configured set of Prometheus alert rules by default.

To ensure you are referencing the latest default alert rules, check the source file of alert definitions in the repository’s prometheus_alerts.yaml file.

Default alert rules

Alert	Severity	Notes
OpenSearchScrapeFailed		Triggered when the prometheus scrape fails.
OpenSearchClusterRed		Triggered when the health status of the cluster is red, meaning that principal shards are not allocated.
OpenSearchClusterYellowTemp		Triggered when shards are still reallocating or initializing.
OpenSearchClusterYellow		Triggered when some replicas shards are unassigned. Might require scale the application to host all shards
OpenSearchWriteRequestsRejectionJumps		Triggered when the write request rejection is bigger than 5%. Might indicate that the node may not keep up with the indexing speed.
OpenSearchNodeDiskLowWatermarkReached		Triggered when disks reach 85% of the capacity.
OpenSearchNodeDiskHighWatermarkReached		Triggered when disks reach 90% of the capacity.
OpenSearchJVMHeapUseHigh		Triggered when the JVM Heap usage in a node reaches 75%.
OpenSearchHostSystemCPUHigh		Triggered when system CPU usage in a node reaches 90%.
OpenSearchProcessCPUHigh		Triggered when process CPU usage in a node reaches 90%.
OpenSearchThrottling		Triggered when a cluster is throttling. Might indicate that is necessary to review indexing request rate, index lifecycle or scale the application.
OpenSearchThrottlingTooLong		Triggered when a cluster is constantly throttling for at least 20 minutes. Might indicate that is necessary to review indexing request rate, index lifecycle or scale the application.

Logs

All the logs from the OpenSearch payload are available in the Grafana web interface at Home > Explore

To get OpenSearch logs, go to the Label filters field and set to juju_application = opensearch, select one operation, e.g. Line contains and run the query.

See also: How to connect to the Grafana web interface