4

I know how to delete Kafka connector as mentioned here Kafka Connect - How to delete a connector

But I am not sure if it also delete/erase specific connector related configs, offsets and status from *.sorage.topic for that worker?

For e.g: Lets say I delete a connector having connector-name as"connector-abc-1.0.0" and Kafka connect worker was started with following config.

offset.storage.topic=<topic.name>.internal.offsets
config.storage.topic=<topic.name>.internal.configs
status.storage.topic=<topic.name>.internal.status
  • Now after DELETE call for that connector, will it erased all records from above internal topics for that specific connector?
  • So that I can create new connector with "same name" on same worker but different config(different offset.start or connector.class)?
suraj_fale
  • 978
  • 2
  • 21
  • 53

2 Answers2

13

When you delete a connector, the offsets are retained in the offsets topic. If you recreate the connector with the same name, it will re-use the offsets from the previous execution (even if the connector was deleted in between).

Robin Moffatt
  • 30,382
  • 3
  • 65
  • 92
  • What do you mean by offsets topic? do you mean `__consumer_offsets` or `offset.storage.topic`? ( for sink connectors) – Zvi Mints Aug 16 '21 at 08:04
  • 2
    How can we delete the offset of a connector so that we can recreate a new connector with same name? – Shubham Saroj Jan 28 '22 at 12:43
  • 1
    @ZviMints sink connectors always use consumer group offsets. Source connectors can optionally choose to use the offsets storage topic – OneCricketeer Nov 17 '22 at 13:45
5

Since Kafka is append only, then only way the messages in those Connect topics would be removed is if it were published with the connector name as the message key, and null as the value.

You could inspect those topics using console consumer to see what data is in them including --property print.key=true, and keep the consumer running when you delete a connector.

You can PUT a new config at /connectors/{name}/config, but any specific offsets that are used are dependent upon the actual connector type (sink / source); for example, there is the internal Kafka __consumer_offsets topic, used by Sink connectors, as well as the offset.storage.topic, optionally used by source connectors.

"same name" on same worker but different config(different offset.start or connector.class)?

I'm not sure changing connector.class would be a good idea with the above in mind since it'd change the connector behavior completely. offset.start isn't a property I'm aware of, so you'll need to see the documentation of that specific connector class to know what it does.

OneCricketeer
  • 179,855
  • 19
  • 132
  • 245