Apache Kafka

Channel Revision Published Runs on
3/stable 185 23 Oct 2024
Ubuntu 22.04
3/candidate 195 Yesterday
Ubuntu 22.04
3/beta 194 03 Dec 2024
Ubuntu 22.04
3/edge 193 26 Nov 2024
Ubuntu 22.04
juju deploy kafka --channel 3/edge
Show information

Platform:

Ubuntu
22.04

Configuration backup and restore

Charmed Apache Kafka’s configuration is distributed using Charmed Apache ZooKeeper. A Charmed Apache ZooKeeper backup can be stored on any S3-compatible storage. S3 access and configurations are managed with the s3-integrator charm.

This guide contains step-by-step instructions on how to deploy and configure the s3-integrator charm for AWS S3, send the configurations to the Charmed Apache ZooKeeper application, and finally manage your Charmed Apache ZooKeeper backups.

Configure s3-integrator

First, deploy the s3-integrator charm:

juju deploy s3-integrator
juju run s3-integrator/leader sync-s3-credentials access-key=<aws-access-key-id> secret-key=<aws-secret-key>

Then, use juju config to add your configuration parameters. For example:

juju config s3-integrator \
    endpoint="https://s3.us-west-2.amazonaws.com" \
    bucket="zk-backups-bucket-1" \
    path="/zk-backups" \
    region="us-west-2"

The only mandatory configuration parameter in the command above is the bucket.

Integrate with Charmed Apache ZooKeeper

To pass these configurations to Charmed Apache ZooKeeper, integrate the two applications:

juju integrate s3-integrator zookeeper

You can create, list, and restore backups now:

juju run zookeeper/leader create-backup
juju run zookeeper/leader list-backups
juju run zookeeper/leader restore backup-id=<backup-id-here>

Create a backup

Check that Charmed Apache ZooKeeper deployment with configurations set for S3 storage is active and idle with the juju status command. Once it’s active, create a backup with the create-backup command:

juju run zookeeper/leader create-backup

Charmed Apache ZooKeeper backups created with the command above will always be full backups: a copy of all the Charmed Apache Kafka configuration will be stored in S3.

The command will output the ID of the newly created backup:

                                     Backup created
┏━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃ Id                   ┃ Log-sequence-number ┃ Path                           ┃
┡━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩
│ 2024-09-12T14:32:46Z │          8589934621 │ zookeeper_backups/2024-09-12T1 │
│                      │                     │ 4:32:46Z/snapshot              │
└──────────────────────┴─────────────────────┴────────────────────────────────┘

List backups

To list available backups, run the list-backups command:

juju run zookeeper/leader list-backups

This command shows available backups, for example:

                                     Backups
┏━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃ Id                   ┃ Log-sequence-number ┃ Path                           ┃
┡━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩
│ 2024-09-12T14:32:46Z │          8589934621 │ zookeeper_backups/2024-09-12T1 │
│                      │                     │ 4:32:46Z/snapshot              │
│ 2024-09-12T14:32:00Z │          8589934621 │ zookeeper_backups/2024-09-12T1 │
│                      │                     │ 4:32:00Z/snapshot              │
│ 2024-09-12T14:26:12Z │          8589934621 │ zookeeper_backups/2024-09-12T1 │
│                      │                     │ 4:26:12Z/snapshot              │
└──────────────────────┴─────────────────────┴────────────────────────────────┘

Below is a list of parameters shown for each backup:

  • Id: identifier of the backup.
  • Log-Sequence-number: a database-specific number to identify its state. Learn more about the Zxid on Apache ZooKeeper documentation.
  • Path: path of the snapshot file in the S3 repository.

Restore a backup

This operation puts you at risk of losing unsaved configuration data. We recommend creating a backup first.

To restore from backup, run the restore command and pass the backup-id (in the YYYY-MM-DDTHH:MM:SSZ format) that is listed in the list-backups action output:

juju run zookeeper/leader restore backup-id=<backup-id-here>

The restore will then proceed. Follow its progress using juju status.