How to enable CROSS-REGIONAL asynchronous replication

TODO: Can it be called an “async replica set”? > MySQL calls it “cluster set”, there are other terms. Patroni calls it “multi-cluster” setup

This guide will show you how to set up and manage an asynchronous replica set using an example with two clusters - Rome and Lisbon.

Prerequisites

juju 3.4.2+

Summary

• Set up asynchronous replication
  • Deploy
  • Offer on primary
  • Consume on secondary
  • Promote/switchover a cluster
  • Scale a cluster
• Integrate with client applications
  • Configure database endpoints
  • Internal
  • External
• Manage separated clusters
  • Recover
  • Detach
  • Reuse a detached cluster
  • Remove a detached cluster

Set up asynchronous replication

When PostgreSQL is deployed within a single juju model, replication is performed synchronously. To create an asynchronous replica set, you must deploy two or more PostgreSQL clusters in separate juju models.

TODO: Technical correctness - 1 postgresql cluster per juju model? > Yes

Unique to PostgreSQL: 2 units in one cluster - content on one unit gets sync replicated to other. With 3+ units, one gets replicated synchronously and the other one async! To not depend on OK from sync units (in case they’re lagging or something) : # sync = ceil(#reps/2)

This async is specifically cross-regional. Focused on disaster recovery.

Done manually on pgbouncer for now (remove relation, etc)

Deploy

To deploy two clusters in different servers, create two juju models - one for the rome cluster, one for the lisbon cluster. In the example below, we use the config flag profile=testing to limit memory usage.

juju add-model rome
juju add-model lisbon

juju switch rome
juju deploy postgresql db1 --channel=14/edge/async-replication --config profile=testing --base ubuntu@22.04

juju switch lisbon
juju deploy postgresql db2 --channel=14/edge/async-replication --config profile=testing --base ubuntu@22.04

Offer

Offer asynchronous replication in one of the clusters.

juju switch rome # must be on corresponding cluster
juju offer db1:async-primary async-primary

Consume

Consume asynchronous replication on the other cluster.

juju switch lisbon
juju consume rome.async-primary
juju integrate async-primary db2:async-replica

Promote or switchover a cluster

TODO: Why does rome need to be promoted - is offering not enough to classify it as primary? When is force-promotion necessary? Juju won’t be able to remove relation if a disaster happens/no reply - so instead we enable users to switch the primary cluster. Offering doesn’t define primary.

To define the primary cluster, use the promote-cluster action.

juju run -m rome db1/leader promote-cluster

To switchover and use lisbon as the primary instead, run

juju run -m lisbon db2/leader promote-cluster force-promotion=true

Future plans: failover to replace force-promotion in a situation where clusters are not healthy

Scale a cluster

The two clusters work independently, which means that it’s possible to scale each cluster independently without much hassle. For example:

juju add-unit db1 -n 2 -m rome
juju add-unit db2 -n 2 -m lisbon

TODO: Can you run these commands from any model? > -m defines aim of action, so yes

Integrate with client applications

To make your database available to a client application, you must first offer and consume database endpoints.

TODO: Why add-model app? Is it in preparation for deploying a client app? Can it be deployed in rome or lisbon instead? > Yes it can. But best practice would be to have client separate from other models in case one fails

juju switch rome
juju offer db1:database db1database

juju switch lisbon
juju offer db2:database db2database

juju add-model app ; juju switch app
juju consume rome.db1database
juju consume lisbon.db2database

Internal application

TODO

juju switch app

juju deploy postgresql-test-app
juju deploy pgbouncer --channel 1/stable

juju relate postgresql-test-app:first-database pgbouncer
juju relate pgbouncer db1database

External application

TODO

juju switch app

juju deploy data-integrator --config database-name=mydatabase
juju deploy pgbouncer pgbouncer-external --channel 1/stable

juju relate data-integrator pgbouncer-external
juju relate pgbouncer-external db1database

juju run data-integrator/leader get-credentials

Manage separated clusters

TODO: Better title

Recover a cluster

If the relation between clusters was removed and one side went into a detached or blocked state, relate both clusters again and call the promote-cluster action to restore async replication.

If a cluster has lost a clusterand needs a new one, deploy a new db application and initiate async replication. The data will be copied automatically after the promote-cluster action is called, and the new cluster will join the async replica set.

TODO: What constitutes as “initiating async replication”? Deploy + offer/consume?

Detach a cluster

Clusters in an async replica set can be detached. The detached cluster can then be either removed or reused.

Assuming lisbon is the current primary and we want to detach rome (either for removal or reuse):

juju remove-relation -m lisbon async-primary db2:async-replica

The command above will move the rome cluster into the detached state blocked keeping all the data in place.

TODO: Is the juju state blocked? Is the detached-ness mentioned somewhere in juju status? > Yes state will be blocked and message will say “cluster in read-only mode”, needs action from user to turn it into new cluster (e.g. promote)

Reuse a detached cluster

The following command creates a new cluster in the replica set from the detached rome cluster, keeping its existing data in use:

juju run -m rome db1/leader promote-cluster

Remove a detached cluster

The following command removes the detached rome cluster and destroys its stored data:

juju remove-application -m rome db1 --destroy-storage