0

How does Kafka deal with multiple versions of the same connector plugin provided in the CLASSPATH? For example, let's say I put both mongo-kafka-1.0.0-all.jar and mongo-kafka-1.1.0-all.jar into the respective directory, in order to allow using both versions, depending on what's needed. Unfortunately, the docs do not give away a way to specify the version of connector.class, I can only assume this is dealt with like classloading is usually dealt with Java-wise.

OneCricketeer
  • 179,855
  • 19
  • 132
  • 245
kopaka
  • 535
  • 4
  • 17
  • Yes, the jars are loaded via the Classloader. Is there a specific reason why you need both jars? – OneCricketeer Nov 12 '20 at 16:33
  • @OneCricketeer The two jars were maybe a bad example, as they should be backwards compatible anyway. But if two dev teams work on the same cluster and require different (not compatible) versions of the same connector plugin to connect the Kafka cluster to different systems, this requirement arises. – kopaka Nov 12 '20 at 16:41
  • My suggestion would be to use configuration management tools to prevent that situation from happening. The cluster will need rebooted anyway to pick up new jars, so those two teams would therefore be halting each other's processes during any upgrades. So, you can either form a centralized infrastructure process responsible for the deployment of new/non conflicting connectors, or provide Connect clusters for each team – OneCricketeer Nov 12 '20 at 18:13
  • But if both jars are/should be available from the initial deployment, this does not help. I was thinking about separate clusters as a last resort, apparently that's the only solution for this specific use case. Thanks! – kopaka Nov 13 '20 at 15:03

1 Answers1

0

If you have the same connector plugin that shares the same connector class (e.g. io.confluent.connect.jdbc.JdbcSinkConnector) and you want separate versions of that same connector JAR, you would need to run multiple Kafka Connect workers.

If you have different connectors that use different dependent JARs then this is handled by Kafka Connect's classpath isolation and the plugin.path setting.

Robin Moffatt
  • 30,382
  • 3
  • 65
  • 92