--- title: Charmhub | Deploy Apache Kafka Connect using Charmhub - The Open Operator Collection description: Deploy the latest version of Apache Kafka Connect on any cloud. url: https://charmhub.io/kafka-connect/docs/t-relate-kafka --- # Apache Kafka Connect [Canonical](https://charmhub.io/publisher/data-platform "View all packages from Canonical") * [Canonical](https://charmhub.io/publisher/data-platform "View all packages from Canonical") Platform: 24.04 22.04 4/stable 37 ``` juju deploy kafka-connect --channel 4/stable ``` [Learn to deploy on juju >](https://juju.is/docs/juju/manage-applications) --- #### Relevant links * [Homepage](https://github.com/canonical/kafka-connect-operator) --- #### Contacts ##### Maintainers + [Canonical Data Platform](mailto:data-platform@lists.launchpad.net) * [Submit a bug](https://github.com/canonical/kafka-connect-operator/issues) --- Share your thoughts on this charm with the community on discourse. [Join the discussion](https://discourse.charmhub.io/) This is part of the [Charmed Apache Kafka Tutorial](https://charmhub.io/kafka-connect/docs/t-overview). Please refer to this page for more information and an overview of the content. ## [Integrate with client applications](https://charmhub.io/kafka-connect/docs/t-relate-kafka#p-25436-integrate-with-client-applications) As mentioned in the previous section of the Tutorial, the recommended way to create and manage users is by means of another charm: the [Data Integrator Charm](https://charmhub.io/data-integrator). This lets us to encode users directly in the Juju model, and - as shown in the following - rotate user credentials with and without application downtime using Relations. Relations, or what Juju documentation describes also as [Integrations](https://juju.is/docs/sdk/integration), let two charms to exchange information and interact with one another. Creating a relation between Apache Kafka and the Data Integrator will automatically generate a username, password, and assign read/write permissions on a given topic. This is the simplest method to create and manage users in Charmed Apache Kafka. ### [Data Integrator charm](https://charmhub.io/kafka-connect/docs/t-relate-kafka#p-25436-data-integrator-charm) The [Data Integrator charm](https://charmhub.io/data-integrator) is a bare-bones charm for central management of database users, providing support for different kinds of data platforms (e.g. MongoDB, MySQL, PostgreSQL, Apache Kafka, OpenSearch, etc.) with a consistent, opinionated and robust user experience. To deploy the Data Integrator charm we can use the command `juju deploy` we have learned above: ``` juju deploy data-integrator --channel stable --config topic-name=test-topic --config extra-user-roles=producer,consumer ``` The expected output: ``` Located charm "data-integrator" in charm-hub, revision 11 Deploying "data-integrator" from charm-hub charm "data-integrator", revision 11 in channel stable on jammy ``` ### [Relate to Charmed Apache Kafka](https://charmhub.io/kafka-connect/docs/t-relate-kafka#p-25436-relate-to-charmed-apache-kafka) Now that the Database Integrator charm has been set up, we can relate it to Charmed Apache Kafka. This will automatically create a username, password, and database for the Database Integrator charm. Relate the two applications with: ``` juju relate data-integrator kafka ``` Wait for `juju status --watch 1s` to show: ``` Model Controller Cloud/Region Version SLA Timestamp tutorial overlord localhost/localhost 3.1.6 unsupported 10:04:50Z App Version Status Scale Charm Channel Rev Exposed Message data-integrator active 1 data-integrator stable 11 no kafka active 3 kafka 3/stable 147 no zookeeper active 5 zookeeper 3/stable 114 no Unit Workload Agent Machine Public address Ports Message data-integrator/0* active idle 8 10.244.26.4 kafka/0 active idle 5 10.244.26.43 machine system settings are not optimal - see logs for info kafka/1* active idle 6 10.244.26.6 machine system settings are not optimal - see logs for info kafka/2 active idle 7 10.244.26.19 machine system settings are not optimal - see logs for info zookeeper/0 active idle 0 10.244.26.251 zookeeper/1 active idle 1 10.244.26.129 zookeeper/2 active idle 2 10.244.26.121 zookeeper/3* active idle 3 10.244.26.28 zookeeper/4 active idle 4 10.244.26.174 Machine State Address Inst id Series AZ Message 0 started 10.244.26.251 juju-f1a2cd-0 jammy Running 1 started 10.244.26.129 juju-f1a2cd-1 jammy Running 2 started 10.244.26.121 juju-f1a2cd-2 jammy Running 3 started 10.244.26.28 juju-f1a2cd-3 jammy Running 4 started 10.244.26.174 juju-f1a2cd-4 jammy Running 5 started 10.244.26.43 juju-f1a2cd-5 jammy Running 6 started 10.244.26.6 juju-f1a2cd-6 jammy Running 7 started 10.244.26.19 juju-f1a2cd-7 jammy Running 8 started 10.244.26.4 juju-f1a2cd-8 jammy Running ``` To retrieve information such as the username, password, and topic. Enter: ``` juju run data-integrator/leader get-credentials ``` This should output something like: ``` kafka: consumer-group-prefix: relation-6- endpoints: 10.244.26.43:9092,10.244.26.6:9092,10.244.26.19:9092 password: ILg8C5msYRvqOnGATeFPyw2DKHncritf tls: disabled topic: test-topic username: relation-6 zookeeper-uris: 10.244.26.121:2181,10.244.26.129:2181,10.244.26.174:2181,10.244.26.251:2181,10.244.26.28:2181/kafka ok: "True" ``` Save the value listed under `bootstrap-server`, `username` and `password`. *(Note: your hostnames, usernames, and passwords will likely be different.)* ### [Produce/consume messages](https://charmhub.io/kafka-connect/docs/t-relate-kafka#p-25436-produceconsume-messages) We will now use the username and password to produce some messages to Apache Kafka. To do so, we will first deploy the [Apache Kafka Test App](https://charmhub.io/kafka-test-app): a test charm that also bundles some python scripts to push data to Apache Kafka, e.g.: ``` juju deploy kafka-test-app -n1 --channel edge ``` Once the charm is up and running, you can log into the container ``` juju ssh kafka-test-app/0 /bin/bash ``` and make sure that the Python virtual environment libraries are visible: ``` export PYTHONPATH="/var/lib/juju/agents/unit-kafka-test-app-0/charm/venv:/var/lib/juju/agents/unit-kafka-test-app-0/charm/lib" ``` Once this is set up, you should be able to use the `client.py` script that exposes some functionality to produce and consume messages. You can explore the usage of the script ``` python3 -m charms.kafka.v0.client --help usage: client.py [-h] [-t TOPIC] [-u USERNAME] [-p PASSWORD] [-c CONSUMER_GROUP_PREFIX] [-s SERVERS] [-x SECURITY_PROTOCOL] [-n NUM_MESSAGES] [-r REPLICATION_FACTOR] [--num-partitions NUM_PARTITIONS] [--producer] [--consumer] [--cafile-path CAFILE_PATH] [--certfile-path CERTFILE_PATH] [--keyfile-path KEYFILE_PATH] [--mongo-uri MONGO_URI] [--origin ORIGIN] Handler for running an Apache Kafka client options: -h, --help show this help message and exit -t TOPIC, --topic TOPIC Kafka topic provided by Kafka Charm -u USERNAME, --username USERNAME Kafka username provided by Kafka Charm -p PASSWORD, --password PASSWORD Kafka password provided by Kafka Charm -c CONSUMER_GROUP_PREFIX, --consumer-group-prefix CONSUMER_GROUP_PREFIX Kafka consumer-group-prefix provided by Kafka Charm -s SERVERS, --servers SERVERS comma delimited list of Kafka bootstrap-server strings -x SECURITY_PROTOCOL, --security-protocol SECURITY_PROTOCOL security protocol used for authentication -n NUM_MESSAGES, --num-messages NUM_MESSAGES number of messages to send from a producer -r REPLICATION_FACTOR, --replication-factor REPLICATION_FACTOR replcation.factor for created topics --num-partitions NUM_PARTITIONS partitions for created topics --producer --consumer --cafile-path CAFILE_PATH --certfile-path CERTFILE_PATH --keyfile-path KEYFILE_PATH --mongo-uri MONGO_URI --origin ORIGIN ``` Using this script, you can therefore start producing messages (change the values of `username`, `password` and `servers`) ``` python3 -m charms.kafka.v0.client \ -u relation-6 -p S4IeRaYaiiq0tsM7m2UZuP2mSI573IGV \ -t test-topic \ -s "10.244.26.43:9092,10.244.26.6:9092,10.244.26.19:9092" \ -n 10 --producer \ -r 3 --num-partitions 1 ``` and consume them ``` python3 -m charms.kafka.v0.client \ -u relation-6 -p S4IeRaYaiiq0tsM7m2UZuP2mSI573IGV \ -t test-topic \ -s "10.244.26.43:9092,10.244.26.6:9092,10.244.26.19:9092" \ --consumer \ -c "cg" ``` ### [Charm client applications](https://charmhub.io/kafka-connect/docs/t-relate-kafka#p-25436-charm-client-applications) Actually, the Data Integrator is only a very special client charm, that implements the `kafka_client` relation for exchanging data with Charmed Apache Kafka and user management via relations. For example, the steps above for producing and consuming messages to Apache Kafka have also been implemented in the `kafka-test-app` charm (that also implements the `kafka_client` relation) providing a fully integrated charmed user experience, where producing/consuming messages can simply be achieved using relations. #### Producing messages To produce messages to Apache Kafka, we need to configure the `kafka-test-app` to act as a producer, publishing messages to a specific topic: ``` juju config kafka-test-app topic_name=test_kafka_app_topic role=producer num_messages=20 ``` To start producing messages to Apache Kafka, we **JUST** simply relate the Apache Kafka Test App with Apache Kafka ``` juju relate kafka-test-app kafka ``` This will both take care of creating a dedicated user (as much as done for the data-integrator) as well as start a producer process publishing messages to the `test_kafka_app_topic` topic, basically automating what was done before by hand. After some time, the `juju status` output should show ``` Model Controller Cloud/Region Version SLA Timestamp tutorial overlord localhost/localhost 3.1.6 unsupported 18:58:47+02:00 App Version Status Scale Charm Channel Rev Address Exposed Message ... kafka-test-app active 1 kafka-test-app edge 8 10.152.183.60 no Topic test_kafka_app_topic enabled with process producer ... Unit Workload Agent Address Ports Message ... kafka-test-app/0* active idle 10.1.36.88 Topic test_kafka_app_topic enabled with process producer ... ``` indicating that the process has started. To make sure that this is indeed the case, you can check the logs of the process: ``` juju exec --application kafka-test-app "tail /tmp/*.log" ``` To stop the process (although it is very likely that the process has already stopped given the low number of messages that were provided) and remove the user, you can just remove the relation ``` juju remove-relation kafka-test-app kafka ``` #### Consuming messages Note that the `kafka-test-app` charm can also similarly be used to consume messages by changing its configuration to ``` juju config kafka-test-app topic_name=test_kafka_app_topic role=consumer consumer_group_prefix=cg ``` After configuring the Apache Kafka Test App, just relate it again with the Charmed Apache Kafka. This will again create a new user and start the consumer process. ## [What’s next?](https://charmhub.io/kafka-connect/docs/t-relate-kafka#p-25436-whats-next) In the next section, we will learn how to rotate and manage the passwords for the Apache Kafka users, both the admin one and the ones managed by the Data Integrator. ---