1

I come across two phrases with respect to ordering,

  1. Messages sent by a producer to a particular topic partition will be appended in the order they are sent. That is, if a record M1 is sent by the same producer as a record M2, and M1 is sent first, then M1 will have a lower offset than M2 and appear earlier in the log.

Another

  1. (config param) max.in.flight.requests.per.connection - The maximum number of unacknowledged requests the client will send on a single connection before blocking. Note that if this setting is set to be greater than 1 and there are failed sends, there is a risk of message re-ordering due to retries (i.e., if retries are enabled).

The question is, will the order still be retained to a particular partition if there are failed sends like mentioned #2 ? if there is a potential issue with one message , all the following messages will be dropped "to retain the order" per partition or the "correct" messages will be sent and failed messages will be notified to the application ?

Michael Heil
  • 16,250
  • 3
  • 42
  • 77
Nag
  • 1,818
  • 3
  • 24
  • 41

1 Answers1

3

"will the order still be retained to a particular partition if there are failed sends like mentioned #2?"

As written in the documentation part you have copied, there is a risk that the ordering is changed.

Imagine, you have a topic with e.g. one partition. You set the retries to 100 and the max.in.flight.requests.per.connection to 5 which is greater than one. As a note, retries will only make sense if you set the acks to 1 or "all".

If you plan to produce the following messages in the order K1, K2, K3, K4, K5 and it takes your producer some time to

  • actually create the batch and
  • make a request to the broker and
  • wait for the acknowledgement of the broker

you could have up to 5 requests in parallel (based on the setting of max.in.flight.request.per.connection). Now, producing "K3" has some issues and it goes into the retry-loop, the messages K4 and K5 can be produced as the request was already in flight.

Your topic would end up with messages in that order: K1, K2, K4, K5, K3.

In case you enable idempotency in the Kafka Producer, the ordering would still be guaranteed as explained in Ordering guarantees when using idempotent Kafka Producer

Michael Heil
  • 16,250
  • 3
  • 42
  • 77
  • hmm, which means the "said" statement doesn't hold good in all aspects which is kind of contradicting the statement - "At a high-level Kafka gives the following guarantees" perhaps with some term and conditions applied . – Nag May 16 '20 at 07:37
  • hehe, true :) I think the devil is in the part "in the order they are **sent**". "Sent" here means: batching, sending to broker and waiting for acknowledgment from broker, which is not obvious when reading... Especially because the default for `max.in.flight.requests.per.connection` is actually 5. – Michael Heil May 16 '20 at 08:44
  • 1
    Maybe worth mentioning idempotence? – OneCricketeer Apr 24 '21 at 16:20