Java Kafka console-consumer running out of heap space

Question

I am running the following command in Console. kafka-console-consumer and receiving out of memory error: java.lang.OutOfMemoryError: Java heap space. How can I resolve this? Trying to apply this resource, except I do not have a topics file.

Kafka Connect running out of heap space

Console:

kafka-console-consumer --bootstrap-server localhost:9092 --topic encounter.personEvent

Application.Yml

kafka:
  topic: encounter
  event: personEvent

Error Message below:

ERROR Error processing message, terminating consumer process:  (kafka.tools.ConsoleConsumer$)
java.lang.OutOfMemoryError: Java heap space
    at java.base/java.nio.HeapByteBuffer.<init>(HeapByteBuffer.java:64)
    at java.base/java.nio.ByteBuffer.allocate(ByteBuffer.java:363)
    at org.apache.kafka.common.memory.MemoryPool$1.tryAllocate(MemoryPool.java:30)
    at org.apache.kafka.common.network.NetworkReceive.readFrom(NetworkReceive.java:113)
    at org.apache.kafka.common.network.KafkaChannel.receive(KafkaChannel.java:452)
    at org.apache.kafka.common.network.KafkaChannel.read(KafkaChannel.java:402)
    at org.apache.kafka.common.network.Selector.attemptRead(Selector.java:674)
    at org.apache.kafka.common.network.Selector.pollSelectionKeys(Selector.java:576)
    at org.apache.kafka.common.network.Selector.poll(Selector.java:481)
    at org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:551)
    at org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.poll(ConsumerNetworkClient.java:265)
    at org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.poll(ConsumerNetworkClient.java:236)
    at org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.poll(ConsumerNetworkClient.java:215)
    at org.apache.kafka.clients.consumer.internals.AbstractCoordinator.ensureCoordinatorReady(AbstractCoordinator.java:246)
    at org.apache.kafka.clients.consumer.internals.ConsumerCoordinator.poll(ConsumerCoordinator.java:480)
    at org.apache.kafka.clients.consumer.KafkaConsumer.updateAssignmentMetadataIfNeeded(KafkaConsumer.java:1262)
    at org.apache.kafka.clients.consumer.KafkaConsumer.poll(KafkaConsumer.java:1231)
    at org.apache.kafka.clients.consumer.KafkaConsumer.poll(KafkaConsumer.java:1211)
    at kafka.tools.ConsoleConsumer$ConsumerWrapper.receive(ConsoleConsumer.scala:454)
    at kafka.tools.ConsoleConsumer$.process(ConsoleConsumer.scala:101)
    at kafka.tools.ConsoleConsumer$.run(ConsoleConsumer.scala:75)
    at kafka.tools.ConsoleConsumer$.main(ConsoleConsumer.scala:52)
    at kafka.tools.ConsoleConsumer.main(ConsoleConsumer.scala)

OneCricketeer · Accepted Answer · 2021-09-30T18:39:51.733

3

KAFKA_HEAP_OPTS is a shared shell variable across all Kafka CLI scripts, so the answer you're referring to in the linked post is correct.

e.g

KAFKA_HEAP_OPTS='-Xmx1G' kafka-console-consumer --bootstrap-server localhost:9092 --topic encounter.personEvent

my apis are running on localhost:8080/swagger-ui.html. and call Kafka producer event

Then you want an HTTP client to POST an HTTP-request to localhost:8080. That, in turn, generates a Kafka message as you say, which is sent to localhost:9092 (by default). You need to consume from Kafka, not an HTTP server

edited Sep 30 '21 at 18:39

answered Sep 30 '21 at 18:22

OneCricketeer

179,855
19
132
245

@mattsmith5 Because 9092 is the default Kafka port. Why did **you** change it for your server? – OneCricketeer Sep 30 '21 at 18:30
You cannot consume from an HTTP server. You would use `curl localhost:8080` for that – OneCricketeer Sep 30 '21 at 18:30
You can use the Swagger UI, as well. Doesn't really matter, but I suspect you will get the data you want if you use the correct port in your consumer CLI – OneCricketeer Sep 30 '21 at 18:38
`kafka-console-consumer.sh` is just a wrapper around Java/Scala code. You don't need to write it in your own app unless you do need that data there as well. Yes, it's for testing/scripting, but you still need to point it at the correct address (which is not an HTTP server) – OneCricketeer Sep 30 '21 at 18:46

score 1 · Answer 2 · answered Aug 02 '23 at 00:22

OOM java heap-space execption can be due to multiple reasons. In my case (AWS MSK kafka - IAM auth protected), it was due to permission access restriction. I had to mention client.properties file while running kafka-console-consumer.sh script. --consumer.config /path/to/client.properties. see here youtube.com/watch?v=r12HYxWAJLo&t=477s

Java Kafka console-consumer running out of heap space

2 Answers2