52

I've been experimenting with Kafka and saw from the documentation on the main site that you're able to set different options for the jvm like heap size and the garbage collector that it uses:

http://kafka.apache.org/documentation.html#java

What it doesn't say, however, is how/where to set these options. The application comes with a /config directory containing a lot of files used for configuration purposes but none that are for Java. It also comes with a /bin directory containing a bunch of scripts for Kafka but again, nothing really indicating how to configure Java.

So my question is, how do I configure the Java options that Kafka uses? Is it done through a file or is there a different way?

Luis Medina
  • 1,112
  • 1
  • 11
  • 20

6 Answers6

78

Modifying a script in the bin directory is highly unrecommended. When upgrading Kafka to the next version, extracting the new binaries would override the changes made in the script.

The preferred way should be to set the environment variable KAFKA_HEAP_OPTS outside the script.

export KAFKA_HEAP_OPTS="-Xmx1G -Xms1G"

If the var is set before starting Kafka via the script it will use the var instead of the default values defined in /bin/kafka-server-start.sh

Marko Topolnik
  • 195,646
  • 29
  • 319
  • 436
benny.la
  • 1,212
  • 12
  • 26
  • I agree with this. I think back when I had asked this question, I could not find a single piece of documentation explaining how to modify the default values which is why resorting to modifying the startup script seemed like the only option at the time. It seems that is not the case since this can be set outside of the script. Thanks for pointing this out. – Luis Medina Apr 15 '16 at 20:24
  • How do you do the same for Kafka Connect though? – eddyP23 Oct 30 '17 at 10:10
  • 4
    I am a bit blind. After you dig a bit deeper, you find that `connect-distributed` uses `kafka-run-class` that uses exact same `KAFKA_HEAP_OPTS` var. Leaving this here for other blind people like me. – eddyP23 Oct 30 '17 at 10:19
  • I am running Confluent 5.0.0 and the default value is set as "-Xmx1G -Xms1G" `if [ "x$KAFKA_HEAP_OPTS" = "x" ]; then export KAFKA_HEAP_OPTS="-Xmx1G -Xms1G" fi` – Vezir Oct 30 '18 at 14:45
18

Another way to do this is by modifying information written in /bin/kafka-server-start.sh:

export KAFKA_HEAP_OPTS="-Xmx1G -Xms1G"

or in /bin/kafka-run-class.sh:

KAFKA_JVM_PERFORMANCE_OPTS="-server -XX:+UseCompressedOops -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:+CMSClassUnloadingEnabled -XX:+CMSScavengeBeforeRemark -XX:+DisableExplicitGC -Djava.awt.headless=true"
Salvador Dali
  • 214,103
  • 147
  • 703
  • 753
12

Looking at kafka-run-classh.sh - kafka uses the following variables:

  • $KAFKA_HEAP_OPTS
  • $KAFKA_JVM_PERFORMANCE_OPTS
  • $KAFKA_GC_LOG_OPTS
  • $KAFKA_JMX_OPTS
  • $KAFKA_LOG4J_OPTS

You can run it via: 

export KAFKA_HEAP_OPTS="-Xmx1G -Xms1G" 
export KAFKA_JMX_OPTS="-Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.port=12346 -Dcom.sun.management.jmxremote.rmi.port=12346 -Dcom.sun.management.jmxremote.local.only=false -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false" 
bin/kafka-server-start.sh -daemon config/server.properties
OneCricketeer
  • 179,855
  • 19
  • 132
  • 245
mfedko
  • 131
  • 1
  • 6
  • 1
    You can run it via _sudo KAFKA_HEAP_OPTS="-Xmx1G -Xms1G" KAFKA_JMX_OPTS="-Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.port=12346 -Dcom.sun.management.jmxremote.rmi.port=12346 -Dcom.sun.management.jmxremote.local.only=false -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false" bin/kafka-server-start.sh -daemon config/server.properties_ – mfedko Apr 12 '17 at 05:41
  • I like ppl showing where it is used. So if one day they change this var's name, I have a place to look into. Thanks. – WesternGun Feb 19 '18 at 09:21
5

You can also set this parameters via your service definition as shown below.

[Unit]
Requires=zookeeper.service
After=zookeeper.service

[Service]
Type=simple
User=kafka
ExecStart=/bin/sh -c '/home/kafka/kafka/bin/kafka-server-start.sh /home/kafka/kafka/config/server.properties > /home/kafka/kafka.log 2>&1'
ExecStop=/home/kafka/kafka/bin/kafka-server-stop.sh
Restart=on-abnormal
Environment="KAFKA_HEAP_OPTS=-Xmx4G"
Environment="KAFKA_JMX_OPTS=-Dcom.sun.management.jmxremote=true -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false -Djava.net.preferIPv4Stack=true"
Environment="JMX_PORT=9999"

[Install]
WantedBy=multi-user.target
H.Ç.T
  • 3,335
  • 1
  • 18
  • 37
3

To add generic JVM settings (for example user timezone) you could use KAFKA_OPTS environment variable from kafka-run-class.sh:

# Generic jvm settings you want to add
if [ -z "$KAFKA_OPTS" ]; then
  KAFKA_OPTS=""
fi
Alex Rewa
  • 319
  • 4
  • 11
-2

You can pass java parameters from command line . E.g.

java -server -Xms3072m -Xmx3072m -XX:NewSize=256m -XX:MaxNewSize=256m  -classpath <long list of jars> foo.class

For a `production server config you can create a properties file or set them in Code by creating

 Properties props = new Properties();
 props.put("serializer.class", "kafka.serializer.StringEncoder");

And then supply them to producerConfig

ProducerConfig config = new ProducerConfig(props);
sol4me
  • 15,233
  • 5
  • 34
  • 34
  • For the properties file, what would that look like in terms of the file itself and in conjunction with starting up the server? For the code option, where exactly would this code go? This is all assuming that I'm using the kafka-server-start.sh script to start Kafka. – Luis Medina Dec 29 '14 at 01:58
  • @massivedynamic http://www.programcreek.com/java-api-examples/index.php?api=kafka.producer.ProducerConfig – sol4me Dec 29 '14 at 15:31
  • `ProducerConfig` is not what the question is asking about – OneCricketeer May 16 '18 at 05:40