3

I am facing frequent issues related to db history topic which is created by the connector itself. There is a temporary solution (by changing the name of the db history topic) which I tried but it's not the better way to handle it. Also, the retention byte is set to -1. This is the error stack.

ERROR WorkerSourceTask{id=cdcit.ventures.sandbox.streamdomain.streamsubdomain.order-filter-0} Task threw an uncaught and unrecoverable exception (org.apache.kafka.connect.runtime.WorkerTask)
io.debezium.DebeziumException: The db history topic or its content is fully or partially missing. Please check database history topic configuration and re-execute the snapshot.
        at io.debezium.relational.HistorizedRelationalDatabaseSchema.recover(HistorizedRelationalDatabaseSchema.java:47)
        at io.debezium.connector.sqlserver.SqlServerConnectorTask.start(SqlServerConnectorTask.java:87)
        at io.debezium.connector.common.BaseSourceTask.start(BaseSourceTask.java:101)
        at org.apache.kafka.connect.runtime.WorkerSourceTask.execute(WorkerSourceTask.java:213)
        at org.apache.kafka.connect.runtime.WorkerTask.doRun(WorkerTask.java:184)
        at org.apache.kafka.connect.runtime.WorkerTask.run(WorkerTask.java:234)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
[2020-09-04 19:12:26,445] ERROR WorkerSourceTask{id=cdcit.ventures.sandbox.streamdomain.streamsubdomain.order-filter-0} Task is being killed and will not recover until manually restarted (org.apache.kafka.connect.runtime.WorkerTask)
Shivani Bhakal
  • 41
  • 1
  • 2

2 Answers2

1

You must use a single database history topic per connector. The topic must not be used by more than one connector.

Jiri Pechanec
  • 1,816
  • 7
  • 8
  • how can we handle update connector to change database history topic without dataloss or duplication of data ? -- @Jiri Pechanec – user1262479 Jan 06 '21 at 07:42
  • @Jiri Pechanec Is there a way to update the running connectors to change the database history topic without any data loss? – Shivani Bhakal Jan 06 '21 at 07:50
  • 1
    You need to execute schema_only_recovery snapshot. But it is necessary that there is no unprocess schema change in the binlog. – Jiri Pechanec Jan 07 '21 at 09:51
  • 1
    @JiriPechanec what would the process be if we delete a connector, delete all topics associated with the connector (including the history topic) and then recreate the connector with the same name? – Dude0001 Aug 23 '21 at 16:15
0

please change the value of parameter "name" in the config "connector.properties" to a new name. Thanks.

Amir Safardoust
  • 1
  • 1
  • Your answer could be improved with additional supporting information. Please [edit] to add further details, such as citations or documentation, so that others can confirm that your answer is correct. You can find more information on how to write good answers [in the help center](/help/how-to-answer). – Community Nov 08 '22 at 08:14