1

I am trying to specify a topic partition for my Kafka Connect Sink. In particular, I am using the DataStax Apache Kafka Connector.

There is a good amount of documentation and resources related to specifying a topic partition for a Kafka Consumer, for example:

However, I haven't been able to find anything at all regarding how to specify what partition a given Kafka Connect Sink Connector reads from.

It seems like the Confluent connector developer docs imply that specifying partition should be possible, but I don't see any config that I can set in the generic Kafka Sink Configuration Properties docs nor in the DSE Kafka Connector configuration docs.

My understanding is that a Kafka Connect Sink is basically a specific implementation of a Kafka Consumer that writes to a given data store. If so, it should be possible to specify a partition, is that correct? Or am I misunderstanding something about how Kafka Connectors work?

Erick Ramirez
  • 13,964
  • 1
  • 18
  • 23
RyanQuey
  • 705
  • 1
  • 9
  • 29
  • 2
    why do you need to fetch only specific partition? – Alex Ott Aug 27 '21 at 17:03
  • @AlexOtt We're thinking of using a topic topology where we are consuming from a topic, enriching the data, then writing back to the same topic but to different partitions. Kafka Connect would then consume only the enriched data on those specific partitions. I think the idea would be that this topic could be scaled and managed as a single unit. We can probably get around this though by just using two separate topics – RyanQuey Aug 28 '21 at 01:40
  • 1
    It’s not very scalable solution and it’s very easy to break something. Two topics is better – Alex Ott Aug 28 '21 at 05:10

1 Answers1

2

You cannot specify partitions in the Connect API. It subscribes to all partitions, then distributes consumer instances amongst worker tasks as part of a consumer group.

OneCricketeer
  • 179,855
  • 19
  • 132
  • 245