• By Canonical Data Platform
Channel Revision Published Runs on
3/stable 123 24 May 2023
Ubuntu 22.04
3/edge 123 23 May 2023
Ubuntu 22.04
juju deploy kafka --channel 3/stable
Show information



Performance Tuning

This section contains some suggested values to get the better performance from Charmed Kafka.

Virtual Memory Handling - Recommended

Kafka brokers make heavy use of the OS page cache to maintain performance. They never normally explicitly issue a command to ensure messages have been persisted to disk (sync), relying instead on the underlying OS to ensure that larger chunks (pages) of data are persisted from the page cache to the disk when the OS deems it efficient and/or necessary to do so. As such, there are a range of runtime kernel parameter tuning that are recommended to set on machines running Kafka to improve performance.

In order to configure these settings, one can write them to /etc/sysctl.conf using sudo echo $SETTING >> /etc/sysctl.conf. Note that the settings shown below are simply sensible defaults that may not apply to every workload:

# ensures low likelihood of memory being assigned to swap-space rather than drop pages from the page cache

# higher ratio results in less frequent disk flushes and better disk I/O performance

Memory Maps - Recommended

Each Kafka log segment requires an index file and a timeindex file, both requiring 1 map area. The default OS maximum number of memory map areas a process can have is set by vm.max_map_count=65536. For production deployments with a large number of partitions and log-segments, it is likely to exceed the maximum OS limit.

It is recommended to set the mmap number sufficiently higher than the number of memory mapped files. This can also be written to /etc/sysctl.conf:


File Descriptors - Recommended

Kafka uses file descriptors for log segments and open connections. If a broker hosts many partitions, keep in mind that the broker requires at least (number_of_partitions)*(partition_size/segment_size) file descriptors to track all the log segments and number of connections.

In order to configure those limits, update the values and add the following to /etc/security/limits.d/root.conf:

#<domain> <type> <item> <value>
root soft nofile 262144
root hard nofile 1024288

Networking - Optional

If you are expecting a large amount of network traffic, kernel parameter tuning may help meet that expected demand. These can also be written to /etc/sysctl.conf:

# default send socket buffer size
# default receive socket buffer size
# maximum send socket buffer size
# maximum receive socket buffer size
# memory reserved for TCP send buffers
# memory reserved for TCP receive buffers
# TCP Window Scaling option
# maximum number of outstanding TCP connection requests
# maximum number of queued packets on the kernel input side (useful to deal with spike of network requests).

Help us improve this documentation

Most of this documentation can be collaboratively discussed and changed on the respective topic in the doc category of the Charmhub forum. See the documentation guidelines if you’d like to contribute.

Last updated 19 days ago. Help improve this document in the forum.