Kafka

This is part of the Charmed Kafka Tutorial. Please refer to this page for more information and the overview of the content.

Deploy Charmed Kafka (and Charmed Zookeeper)

To deploy Charmed Kafka, all you need to do is run the following commands, which will automatically fetch Kafka and Zookeeper charms from Charmhub and deploy them to your model. For example, to deploy a 5 Zookeeper unit and 3 Kafka unit cluster, you can simply run

$ juju deploy zookeeper -n 5
$ juju deploy kafka -n 3

After this, it is necessary to connect them:

$ juju relate kafka zookeeper

Juju will now fetch Charmed Kafka and Zookeeper and begin deploying it to the LXD cloud. This process can take several minutes depending on how provisioned (RAM, CPU, etc) your machine is. You can track the progress by running:

juju status --watch 1s

This command is useful for checking the status of Charmed Zookeeper and Charmed Kafka and gathering information about the machines hosting the two applications. Some of the helpful information it displays include IP addresses, ports, state, etc. The command updates the status of the cluster every second and as the application starts you can watch the status and messages of Charmed Kafka and Zookeeper change.

Wait until the application is ready - when it is ready, juju status --watch 1s will show:

Model     Controller  Cloud/Region         Version  SLA          Timestamp
tutorial  overlord    localhost/localhost  3.1.6    unsupported  08:20:12Z

App        Version  Status  Scale  Charm      Channel      Rev  Exposed  Message
kafka               active      3  kafka      3/stable     147  no       
zookeeper           active      5  zookeeper  3/stable     114  no       

Unit          Workload  Agent  Machine  Public address  Ports  Message
kafka/0       active    idle   5        10.244.26.43           machine system settings are not optimal - see logs for info
kafka/1*      active    idle   6        10.244.26.6            machine system settings are not optimal - see logs for info
kafka/2       active    idle   7        10.244.26.19           machine system settings are not optimal - see logs for info
zookeeper/0   active    idle   0        10.244.26.251          
zookeeper/1   active    idle   1        10.244.26.129          
zookeeper/2   active    idle   2        10.244.26.121          
zookeeper/3*  active    idle   3        10.244.26.28           
zookeeper/4   active    idle   4        10.244.26.174          

Machine  State    Address        Inst id        Series  AZ  Message
0        started  10.244.26.251  juju-f1a2cd-0  jammy       Running
1        started  10.244.26.129  juju-f1a2cd-1  jammy       Running
2        started  10.244.26.121  juju-f1a2cd-2  jammy       Running
3        started  10.244.26.28   juju-f1a2cd-3  jammy       Running
4        started  10.244.26.174  juju-f1a2cd-4  jammy       Running
5        started  10.244.26.43   juju-f1a2cd-5  jammy       Running
6        started  10.244.26.6    juju-f1a2cd-6  jammy       Running
7        started  10.244.26.19   juju-f1a2cd-7  jammy       Running

To exit the screen with juju status --watch 1s, enter Ctrl+c.

Access Kafka cluster

To watch the process, juju status can be used. Once all the units show as active|idle the credentials to access a broker can be queried with:

juju run kafka/leader get-admin-credentials

The output of the previous command is something like this:

client-properties: |-
  sasl.jaas.config=org.apache.kafka.common.security.scram.ScramLoginModule required username="admin" password="e2sMfYLQg7sbbBMFTx1qlaZQKTUxr09x";
  bootstrap.servers=10.244.26.19:9092,10.244.26.6:9092,10.244.26.43:9092
  security.protocol=SASL_PLAINTEXT
  sasl.mechanism=SCRAM-SHA-512
password: e2sMfYLQg7sbbBMFTx1qlaZQKTUxr09x
username: admin

Providing you the username and password of the Kafka cluster admin user.

IMPORTANT Note that when no other application is related to Kafka, the cluster is secured-by-default and external listeners (binded to port 9092) are disabled, thus preventing any external incoming connection.

Nevertheless, it is still possible to run a command from within the Kafka cluster using the internal listeners in place of the external ones. The internal endpoints can be constructed by replacing the 19092 port in the bootstrap.servers returned in the output above, e.g.

INTERNAL_LISTENERS=$(juju run kafka/leader get-admin-credentials | grep "bootstrap.servers" | cut -d "=" -f2 | sed -s "s/\:9092/:19092/g")

Once you have fetched the INTERNAL_LISTENERS, log in into one of the Kafka container in one of the units

juju ssh kafka/leader sudo -i

When the unit is started, the Charmed Kafka Operator installs the charmed-kafka Snap in the unit that provides a number of entrypoints (that corresponds to the bin commands in the Kafka distribution) for performing various administrative tasks, e.g charmed-kafka.config to update cluster configuration, charmed-kafka.topics for topic management, and many more! Within the machine, the Charmed Kafka Operator also creates a client.properties file that already provides the relevant settings to connect to the cluster using the CLI

CLIENT_PROPERTIES=/var/snap/charmed-kafka/current/etc/kafka/client.properties

For example, in order to create a topic, you can run:

charmed-kafka.topics \
    --create --topic test_topic \
    --bootstrap-server $INTERNAL_LISTENERS \
    --command-config $CLIENT_PROPERTIES

You can similarly then list the topic, using

charmed-kafka.topics \
    --list \
    --bootstrap-server  $INTERNAL_LISTENERS \
    --command-config $CLIENT_PROPERTIES

making sure the topic was successfully created.

You can finally delete the topic, using

charmed-kafka.topics \
    --delete --topic test_topic \
    --bootstrap-server  $INTERNAL_LISTENERS \
    --command-config $CLIENT_PROPERTIES

Other available Kafka bin commands can also be found with:

snap info charmed-kafka

What’s next?

However, although the commands above can run within the cluster, it is generally recommended during operations to enable external listeners and use these for running the admin commands from outside the cluster. To do so, as we will see in the next section, we will deploy a data-integrator charm and relate it to Kafka.