Alert rules

This page contains a markdown version of the alert rules described in the postgresql-k8s-operator repository. The following file(s) are the source of truth:

• src/prometheus_alert_rules/postgresql_rules.yaml
• src/prometheus_alert_rules/pgbouncer_rules.yaml
• src/prometheus_alert_rules/patroni_rules.yaml

This documentation describes the latest alert rule expressions. See the YAML file(s) on listed above if you require an older version.

PostgresqlExporterK8s

Alert	Severity	Notes
PostgresqlDown		PostgresSQL instance is down. If you are not upgrading or configuring cross-region async replication clusters, check for errors in the Loki logs.
PostgresqlRestarted		PostgresSQL instance has restarted. If you are not enabling/disabling TLS or upgrading or configuring cross-region async replication clusters, check for errors in the Loki logs.
PostgresqlExporterError		PostgresSQL instance is showing an exporter error. There may be a buggy query in query.yaml
PostgresqlTableNotAutoVacuumed		A PostgresSQL table in instance is not auto vacuumed. A table has not been auto vacuumed for 7 days.Double-check your VACUUM settings.
PostgresqlTableNotAutoAnalyzed		A PostgresSQL table in instance is not auto analyzed. A table has not been auto analyzed for 7 days.Double-check your AUTOVACUUM ANALYZE settings.
PostgresqlTooManyConnections		PostgresSQL instance is using > 80% of the maximum connections. Consider checking how many connections the client application is opening, or using PgBouncer in front of the database.
PostgresqlNotEnoughConnections		PostgresSQL instance does not have enough connections. PostgreSQL instance should have more connections (> 5). Consider double-checking how many connections the client application is opening and/or using PgBouncer in front of the database.
PostgresqlDeadLocks		PostgresSQL instance has dead locks. See more details with the pg_locks view.
PostgresqlHighRollbackRate		PostgresSQL instance has a high rollback rate instance. The ratio of transactions being aborted compared to committed is > 2 %. This is probably happening due to unoptimized configurations related to commit delay, connections, memory, and WAL files.
PostgresqlCommitRateLow		PostgresSQL instance has a low commit rate. PostgresSQL seems to be processing very few transactions. Check for long-running queries and configuration issues, like insufficient cache size.
PostgresqlLowXidConsumption		PostgresSQL instance shows low XID consumption. PostgresSQL seems to be consuming transaction IDs very slowly. Run ANALYZE to update the optimizer statistics, ensure that query plans are correct, and double-check your VACUUM settings.
PostgresqlHighRateStatementTimeout		PostgresSQL instance shows a high rate of statement timeout. Either tune statement_timeout when sending queries or use EXPLAIN ANALYZE to understand how the queries can be improved.
PostgresqlHighRateDeadlock		PostgresSQL instance shows a high deadlock rate. More details can be obtained through the pg_locks view.
PostgresqlUnusedReplicationSlot		PostgresSQL instance has unused replication slots. Check if a replica is not using any of them before deleting it.
PostgresqlTooManyDeadTuples		PostgresSQL instance has too many dead tuples. Double-check your VACUUM settings.
PostgresqlConfigurationChanged		PostgresSQL instance configuration has changed. PostgresSQL database configuration has changed.
PostgresqlSslCompressionActive		PostgresSQL instance SSL compression is active. Database connections with SSL compression are enabled. This may add significant jitter in replication delay.Replicas should turn off SSL compression via sslcompression=0 in recovery.conf.
PostgresqlTooManyLocksAcquired		PostgreSQL instance has acquired too many locks. If this alert happens frequently, you may need to increase the PostgresSQL setting max_locks_per_transaction.
PostgresqlBloatIndexHigh(>80%)		PostgreSQL instance has a high bloat index (> 80%). An index is bloated.Consider running REINDEX INDEX CONCURRENTLY <index name>;
PostgresqlBloatTableHigh(>80%)		PostgreSQL instance has a high bloat table (> 80%). A table is bloated.Consider running VACUUM {{ $labels.relname }};
PostgresqlInvalidIndex		PostgresSQL instance )= has an invalid index. A table has an invalid index. Consider running DROP INDEX <index name>;

PgbouncerExporterK8s

Alert	Severity	Notes
PgbouncerActiveConnections		PgBouncer instance has > 200 active connections Consider checking the client application responsible for generating those additional connections.
PgbouncerErrors		PgBouncer instance is logging errors. This may be due to a a server restart or an admin typing commands at the PgBouncer console.
PgbouncerMaxConnections		PgBouncer instance has reached max_client_conn. Consider checking how many connections the client application is opening.

PatroniExporterK8s

Alert	Severity	Notes
PatroniPostgresqlDown		Patroni PostgreSQL instance is down. Check for errors in the Loki logs.
PatroniHasNoLeader		Patroni instance has no leader node. A leader node (neither primary nor standby) cannot be found inside a cluster. Check for errors in the Loki logs.