How to deploy Charmed Kafka K8s

IMPORTANT For non-K8s Charmed Kafka, see the Charmed Kafka documentation instead.

To deploy a Charmed Kafka K8s cluster:

1. Set up a Juju Controller
2. Set up a Juju Model
3. Deploy and relate Kafka K8s and ZooKeeper K8s charms.
4. (Optionally) Create an external admin user

In the next subsections, we will cover these steps separately by referring to relevant Juju documentation and providing details on the Charmed Kafka K8s specifics. If you already have a Juju controller and/or a Juju model, you can skip the associated steps.

Juju controller setup

Make sure you have a Juju controller accessible from your local environment using the Juju client snap.

List available controllers:

juju list-controllers

Switch to another controller if needed:

juju switch <controller>

If there are no suitable controllers, create a new one:

juju bootstrap <cloud> <controller>

where <cloud> – the cloud to deploy controller to, e.g., localhost. For more information on how to setup a new cloud, see the How to manage clouds guide in Juju documentation.

Note See the How to manage controllers guide in Juju documentation for more options.

Juju model setup

You can create a new Juju model using

juju add-model <model>

Alternatively, you can switch to any existing Juju model:

juju switch <model>

Make sure that the model is of a correct type (k8s):

juju show-model | yq '.[].type'

Deploy and relate Kafka K8s and ZooKeeper charms

The Kafka and ZooKeeper charms can both be deployed as follows:

juju deploy kafka-k8s --channel 3/stable -n <kafka-units> --trust
juju deploy zookeeper-k8s --channel 3/stable -n <zookeeper-units>

where <kafka-units> and <zookeeper-units> – the number of units to deploy for Kafka and ZooKeeper. We recommend values of at least 3 and 5 respectively.

NOTE The --trust option is needed for the Kafka application to work properly, e.g., use NodePort or juju refresh. For more information about the trust options usage, see the Juju documentation.

Connect ZooKeeper and Kafka by relating/integrating the charms:

juju relate kafka-k8s zookeeper-k8s

Check the status of the deployment:

juju status

Once all the units show active or idle status, the deployment should be complete.

(Optional) Create an external admin users

Charmed Kafka aims to follow the secure by default paradigm. As a consequence, after being deployed the Kafka cluster won’t expose any external listener. In fact, ports are only opened when client applications are related, also depending on the protocols to be used.

NOTE For more information about the available listeners and protocols please refer to this table.

It is however generally useful for most situations to create a first admin user to be used to manage the Kafka cluster (either internally or externally).

To create an admin user, deploy the Data Integrator Charm with extra-user-roles set to admin:

juju deploy data-integrator --channel stable --config topic-name=test-topic --config extra-user-roles=admin

… and relate it to the Kafka charm:

juju relate data-integrator kafka-k8s

To retrieve authentication information, such as the username and password, use:

juju run data-integrator/leader get-credentials