1

org.apache.kafka.common.errors.RecordTooLargeException: There are some messages at [Partition=Offset]: {binlog-0=170421} whose size is larger than the fetch size 1048576 and hence cannot be returned.

Hi, I'm getting the above exception and my apache beam data pipeline fails. I want the kafka reader to ignore message with size more than default size & maybe push it into another topic for logging purposes.

Properties kafkaProps = new Properties();
kafkaProps.setProperty("errors.tolerance", "all");
kafkaProps.setProperty("errors.deadletterqueue.topic.name", "binlogfail");
kafkaProps.setProperty("errors.deadletterqueue.topic.replication.factor", "1");

Tried using the above but still facing record too large exception.

Kafka Connect sink tasks ignore tolerance limits

This link says that the above properties can be used only during conversion or serialization.

Is there some way to solve the problem that I'm facing. Any help would be appreciated.

Michael Heil
  • 16,250
  • 3
  • 42
  • 77
Deepak Vengatesh
  • 21
  • 3
  • Do you have a full stacktrace? If yes, please post it here. I guess it's not a KafkaIO issue and likely related to your broker configuration as @cricket_007 mentioned before. – Alexey Romanenko Mar 03 '20 at 11:02

1 Answers1

0

I want the kafka reader to ignore message with size more than default size

With Beam, I'm not sure you can capture that error and skip it. You would have to go to the raw Kafka Consumer/Producer instances to handle that try-catch logic

& maybe push it into another topic for logging purposes.

That isn't possible without changing the broker settings to first allow larger messages, and then changing your client properties.

errors.* properties are for Kafka Connect APIs, not Consumer/Producer (such as Beam)

Related - How can I send large messages with Kafka (over 15MB)?

OneCricketeer
  • 179,855
  • 19
  • 132
  • 245
  • We are only consuming, producer is not in our control. Is there any way in which in the consumer side through which I can ignore such large messages which rarely comes. This is for a precautionary measure to ignore such events – Deepak Vengatesh Mar 03 '20 at 08:36
  • The producer would be getting the same error. It's the broker that rejects large messages, by default. If the broker has been updated, you need to update the consumer settings, as per the linked post. And again, no, you need access to the poll / process loop within Beam – OneCricketeer Mar 03 '20 at 08:37
  • Thanks a lot. Will check it. – Deepak Vengatesh Mar 03 '20 at 08:54
  • If this answers your question, feel free to use the checkmark next to the post to accept ;) – OneCricketeer Mar 03 '20 at 08:56