Troubleshooting

Ensure you went into the real issue which requires the manual activity. Run juju status and check the list of charm statuses and recommended activities there.

Logs

Please be familiar with Juju logs concepts and learn how to manage Juju logs.

Always check the Juju logs before troubleshooting further:

juju debug-log --replay --tail

Focus on ERRORS (normally there should be none):

juju debug-log --replay | grep -c ERROR

Consider to enable DEBUG log level IF you are troubleshooting wired charm behavior:

juju model-config 'logging-config=<root>=INFO;unit=DEBUG'

The Patroni/PostgreSQL logs are located in workload (see below) container:

> ls -la /var/log/postgresql/
-rw-r--r-- 1 postgres postgres 23863 Sep 15 13:10 patroni.log
-rw------- 1 postgres postgres  2215 Sep 15 12:57 postgresql.log

If backups are enabled, Pgbackrest logs can also be found in the workload container:

> ls -la /var/log/pgbackrest/
-rw-r----- 1 postgres postgres 2949 Sep 18 10:42 all-server.log
-rw-r----- 1 postgres postgres 3219 Sep 18 10:41 test3.patroni-postgresql-k8s-stanza-create.log

Kubernetes

Check the operator architecture first to be familiar with charm and workload containers. Make sure both containers are Running and Ready to continue troubleshooting inside the charm. To describe the running pod, use the following command (where 0 is a Juju unit id). :

kubectl describe pod postgresql-k8s-0 -n <juju_model_name>
...
Containers:
  charm:
    ...
    Image:          jujusolutions/charm-base:ubuntu-22.04
    State:          Running
    Ready:          True
    Restart Count:  0
    ...
  postgresql:
    ...
    Image:          registry.jujucharms.com/charm/kotcfrohea62xreenq1q75n1lyspke0qkurhk/postgresql-image@something
    State:          Running
    Ready:          True
    Restart Count:  0
    ...

Container charm

To enter the charm container, use:

juju ssh postgresql-k8s/0 bash

Here you can make sure pebble is running, the Pebble plan is

root@postgresql-k8s-0:/var/lib/juju# /charm/bin/pebble services
Service          Startup  Current  Since
container-agent  enabled  active   today at 12:29 UTC

root@postgresql-k8s-0:/var/lib/juju# ps auxww
USER         PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
root           1  0.0  0.0 718264 10876 ?        Ssl  12:29   0:00 /charm/bin/pebble run --http :38812 --verbose
root          15  0.6  0.1 778628 59148 ?        Sl   12:29   0:03 /charm/bin/containeragent unit --data-dir /var/lib/juju --append-env PATH=$PATH:/charm/bin --show-log --charm-modified-version 0

Normally you should have no issues here, if you see some, please contact us.
Feel free to improve this document!

Container postgresql (workload)

To enter the workload container, use:

juju ssh --container postgresql postgresql-k8s/0 bash

You can check the list of running processes and Pebble plan:

root@postgresql-k8s-0:/# /charm/bin/pebble services
Service            Startup   Current   Since
metrics_server     enabled   active    today at 12:30 UTC
pgbackrest server  disabled  inactive  -
postgresql         enabled   active    today at 12:29 UTC

root@postgresql-k8s-0:/# ps auxww
USER         PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
root           1  0.0  0.0 718264 10916 ?        Ssl  12:29   0:00 /charm/bin/pebble run --create-dirs --hold --http :38813 --verbose
postgres      14  0.1  0.1 565020 39412 ?        Sl   12:29   0:01 python3 /usr/bin/patroni /var/lib/postgresql/data/patroni.yml
postgres      30  0.0  0.0 1082704 9076 ?        Sl   12:30   0:00 /usr/bin/prometheus-postgres-exporter
postgres      48  0.0  0.0 215488 28912 ?        S    12:30   0:00 /usr/lib/postgresql/14/bin/postgres -D /var/lib/postgresql/data/pgdata --config-file=/var/lib/postgresql/data/pgdata/postgresql.conf --listen_addresses=0.0.0.0 --port=5432 --cluster_name=patroni-postgresql-k8s --wal_level=logical --hot_standby=on --max_connections=100 --max_wal_senders=10 --max_prepared_transactions=0 --max_locks_per_transaction=64 --track_commit_timestamp=off --max_replication_slots=10 --max_worker_processes=8 --wal_log_hints=on
postgres      50  0.0  0.0  70080  7488 ?        Ss   12:30   0:00 postgres: patroni-postgresql-k8s: logger 
postgres      52  0.0  0.0 215592  9136 ?        Ss   12:30   0:00 postgres: patroni-postgresql-k8s: checkpointer 
postgres      53  0.0  0.0 215604  9632 ?        Ss   12:30   0:00 postgres: patroni-postgresql-k8s: background writer 
postgres      54  0.0  0.0 215488 12208 ?        Ss   12:30   0:00 postgres: patroni-postgresql-k8s: walwriter 
postgres      55  0.0  0.0 216052 10928 ?        Ss   12:30   0:00 postgres: patroni-postgresql-k8s: autovacuum launcher 
postgres      56  0.0  0.0 215488  8420 ?        Ss   12:30   0:00 postgres: patroni-postgresql-k8s: archiver 
postgres      57  0.0  0.0  70196  8384 ?        Ss   12:30   0:00 postgres: patroni-postgresql-k8s: stats collector 
postgres      58  0.0  0.0 216032 10476 ?        Ss   12:30   0:00 postgres: patroni-postgresql-k8s: logical replication launcher 
postgres      60  0.0  0.0 218060 21428 ?        Ss   12:30   0:00 postgres: patroni-postgresql-k8s: operator postgres 127.0.0.1(59528) idle

The list of running Pebble services will depends on configured (enabled) COS integration and/or Backup functionality. The Pebble and it’s service postgresql must always be enabled and currently running (the Linux processes pebble, patroni and postgres).

To connect inside the PostgreSQL, check the charm users concept and request admin credentials and use psql:

> juju run postgresql-k8s/leader get-password username=operator
password: 3wMQ1jzfuERvTEds

> juju ssh --container postgresql postgresql-k8s/0 bash

> > psql -h 127.0.0.1 -U operator -d postgres -W
> > Password for user operator: 3wMQ1jzfuERvTEds
>
> > postgres=# \l
> > postgres  | operator | UTF8     | C       | C.UTF-8 | operator=CTc/operator   +
> >           |          |          |         |         | backup=CTc/operator     +
> ...

Continue troubleshooting your DB/SQL related issues from here.

It is NOT recommended to restart services directly as it might create a split brain situation with operator internal state. If you see the problem with a unit, consider to scale-down and re-scale-up to recover the cluster state.

As a last resort, contact us If you cannot determinate the source of your issue. Also, feel free to improve this document!

Installing extra software:

Sometimes it is necessary to install some extra troubleshooting software. Use the common approach:

root@postgresql-k8s-0:/# apt update && apt install less
...
Setting up less (590-1ubuntu0.22.04.1) ...
root@postgresql-k8s-0:/#