Minor Upgrade

Example: PostgreSQL 14.8 -> PostgreSQL 14.9
(including simple charm revision bump: from revision 193 to revision 196).

This is part of the Charmed PostgreSQL Upgrade. Please refer to this page for more information and the overview of the content.

We strongly recommend to NOT perform any other extraordinary operations on Charmed PostgreSQL cluster, while upgrading. As an examples, these may be (but not limited to) the following:

1. Adding or removing units
2. Creating or destroying new relations
3. Changes in workload configuration
4. Upgrading other connected/related/integrated applications simultaneously

The concurrency with other operations is not supported, and it can lead the cluster into inconsistent states.

NOTE: Make sure to have a backup of your data when running any type of upgrades!

TIP: It’s recommended to deploy your application in conjunction with the PgBouncer. This will ensure minimal service disruption, if any.

Minor upgrade steps

1. Collect all necessary pre-upgrade information. It will be necessary for the rollback (if requested). Do NOT skip this step, it is better safe the sorry!
2. Prepare “Charmed PostgreSQL” Juju application for the in-place upgrade. See the step description below for all technical details executed by charm here.
3. Upgrade. Once started all units in a cluster will be executed sequentially. The upgrade will be aborted (paused) if the unit upgrade has failed.
4. (optional) Consider to Rollback in case of disaster. Please inform and include us in your case scenario troubleshooting to trace the source of the issue and prevent it in the future. Contact us!
5. Post-upgrade check. Make sure all units are in the proper state and the cluster is healthy.

Step 1: Collect

NOTE: The step is only valid when deploying from charmhub. If the local charm deployed (revision is small, e.g. 0-10), make sure the proper/current local revision of the .charm file is available BEFORE going further. You might need it for rollback.

The first step is to record the revision of the running application, as a safety measure for a rollback action. To accomplish this, simply run the juju status command and look for the deployed Charmed PostgreSQL revision in the command output, e.g.:

Model        Controller  Cloud/Region         Version  SLA          Timestamp
welcome-lxd  lxd         localhost/localhost  3.1.6    unsupported  11:35:36+02:00

App         Version  Status  Scale  Charm       Channel       Rev  Exposed  Message
postgresql  14.9     active      3  postgresql  14/candidate  330  no       

Unit           Workload  Agent  Machine  Public address  Ports     Message
postgresql/3*  active    idle   3        10.3.217.74     5432/tcp  
postgresql/4   active    idle   4        10.3.217.95     5432/tcp  
postgresql/5   active    idle   5        10.3.217.108    5432/tcp  

Machine  State    Address       Inst id        Base          AZ  Message
3        started  10.3.217.74   juju-d483b7-3  ubuntu@22.04      Running
4        started  10.3.217.95   juju-d483b7-4  ubuntu@22.04      Running
5        started  10.3.217.108  juju-d483b7-5  ubuntu@22.04      Running

For this example, the current revision is 330. Store it safely to use in case of rollback!

Step 2: Prepare

Before running the juju refresh command it’s necessary to run the pre-upgrade-check action against the leader unit (ensure no errors in the result output):

juju run postgresql/leader pre-upgrade-check

The action will configure the charm to minimize the amount of primary switchover, among other preparations for a safe upgrade process. After successful execution, the charm is ready to be upgraded.

Step 3: Upgrade

Use the juju refresh command to trigger the charm upgrade process.

# example with channel selection
juju refresh postgresql --channel 14/edge

# example with specific revision selection
juju refresh postgresql --revision=342

# example with a local charm file
juju refresh postgresql --path ./postgresql_ubuntu-22.04-amd64.charm

All units are going to be refreshed (i.e. receive new charm content), and the upgrade will execute one unit at the time. The juju status will look like*:

Model        Controller  Cloud/Region         Version  SLA          Timestamp
welcome-lxd  lxd         localhost/localhost  3.1.6    unsupported  11:36:18+02:00

App         Version  Status  Scale  Charm       Channel  Rev  Exposed  Message
postgresql  14.9     active      3  postgresql  14/edge  331  no       

Unit           Workload  Agent      Machine  Public address  Ports     Message
postgresql/3*  waiting   idle       3        10.3.217.74     5432/tcp  other units upgrading first...
postgresql/4   waiting   idle       4        10.3.217.95     5432/tcp  other units upgrading first...
postgresql/5   waiting   executing  5        10.3.217.108    5432/tcp  waiting for database initialisation

Machine  State    Address       Inst id        Base          AZ  Message
3        started  10.3.217.74   juju-d483b7-3  ubuntu@22.04      Running
4        started  10.3.217.95   juju-d483b7-4  ubuntu@22.04      Running
5        started  10.3.217.108  juju-d483b7-5  ubuntu@22.04      Running

After each unit completes the upgrade, the message will go blank, and a next unit follow, e.g.:

Model        Controller  Cloud/Region         Version  SLA          Timestamp
welcome-lxd  lxd         localhost/localhost  3.1.6    unsupported  11:36:31+02:00

App         Version  Status  Scale  Charm       Channel  Rev  Exposed  Message
postgresql  14.9     active      3  postgresql  14/edge  331  no       

Unit           Workload     Agent      Machine  Public address  Ports     Message
postgresql/3*  waiting      idle       3        10.3.217.74     5432/tcp  other units upgrading first...
postgresql/4   maintenance  executing  4        10.3.217.95     5432/tcp  refreshing the snap
postgresql/5   active       idle       5        10.3.217.108    5432/tcp  

Machine  State    Address       Inst id        Base          AZ  Message
3        started  10.3.217.74   juju-d483b7-3  ubuntu@22.04      Running
4        started  10.3.217.95   juju-d483b7-4  ubuntu@22.04      Running
5        started  10.3.217.108  juju-d483b7-5  ubuntu@22.04      Running

Note: It is expected to have some status changes during the process: waiting, maintenance, active. Do NOT trigger rollback procedure during the running upgrade procedure. Make sure upgrade has failed/stopped and cannot be fixed/continued before triggering rollback!

Note: Each unit should recover shortly after the upgrade, but the time can vary depending on the amount of data written to the cluster while the unit was not part of the cluster. Please be patient on the huge installations.

After a juju refresh, case there any version incompatibilities in charm revisions or it dependencies, or any other unexpected failure in the upgrade process, the upgrade process will be halted an enter a failure state.

Step 4: Rollback (optional)

The step must be skipped if the upgrade went well!

Although the underlying PostgreSQL Cluster continue to work, it’s important to rollback the charm to previous revision so an update can be later attempted after a further inspection of the failure. Please switch to the dedicated minor rollback tutorial if necessary.

Step 5: Check

The future improvement is planned to check the state on pod/cluster on a low level. At the moment check juju status to make sure the cluster state is OK.