0

We're using KafkaJS to send messages to a Kafka topic that is being consumed by a single consumer. Our Kafka instance is configured in server.properties to only have a single partition, and we can confirm the topic only has a single partition by looking at the output of kafka-topics.sh --describe --bootstrap-server localhost:9093 --topic <topic name>. However, the messages being received by the consumer are out of order. How can this be the case given that our topic only has a single partition?

Caleb Koch
  • 656
  • 1
  • 9
  • 15
  • 1
    Have you checked what order the messages are in on the topic? you can use the kafka-console-consumer.sh for this. – Joe M Nov 16 '22 at 22:24
  • 1
    Kafka will store the messages in the order it receives them in. Could it be possible that KafkaJS is not sending them in sequence? See if you can configure that and also configure acks to 1 or all (on KafkaJS) so that it waits for kafka to acknowledge that it's been written before producing the next message? – Joe M Nov 16 '22 at 22:33
  • you may confirm the order/contents using any gui tools like Kafka topic viewer. – Valath Nov 17 '22 at 00:15

1 Answers1

0

I'm answering my own question here, but it turns out it wasn't an issue with Kafka itself.

The messages were being sent to Kafka inside of a forEach loop. Even though we were making sure to use await with each call to send on each iteration, it turns out forEach doesn't play nice with asynchronous code. The fix was to use a traditional for loop.

Caleb Koch
  • 656
  • 1
  • 9
  • 15