kafka connect not loading manually built connectors

Question

I downloaded aiven-kafka-connect-jdbc and built the jar manually.Placed the same jar in kafka-connect plugin directory and mentioned the plugin path in connect-standalone.properties. But when i run the connect it throws error saying "Failed to find any class that implements Connector and which name matches io.aiven.connect.jdbc.JdbcSourceConnector"

What is missing in my configuration?

connect-standalone.properties

bootstrap.servers=*******:9092
key.converter=org.apache.kafka.connect.json.JsonConverter
value.converter=org.apache.kafka.connect.json.JsonConverter
key.converter.schemas.enable=true
value.converter.schemas.enable=true
offset.storage.file.filename=/Kafka/kafka_2.13-2.8.0/data/kafka/connect/offsets/connect.offsets
offset.flush.interval.ms=10000
plugin.path=/Kafka/kafka_2.13-2.8.0/Plugins,/Kafka/kafka_2.13-2.8.0/libs,

connector.properties

name=aiventransconnector
connector.class=io.aiven.connect.jdbc.JdbcSourceConnector
connection.url=jdbc:oracle:thin:@****&&:1521/****?verifyServerCertificate=false&useSSL=true&requireSSL=true
connection.user=******
connection.password=******
table.whitelist= TRANSACTIONS
tasks.max= 1
auto.create= true
auto.evolve= true
mode=incrementing
incrementing.column.name=TRANSACTION_ID
numeric.mapping=precision_only
topic.prefix=AIVEN

Folder structure of kafka connect plugin

/Kafka/
  -kafka_2.13-2.8.0/
    -Plugins/
      -aiven-kafka-connect-jdbc/
        -lib/
          -aiven-kafka-connect-jdbc-6.7.0-SNAPSHOT.jar with all dependencies

Why did you build instead of download? https://github.com/aiven/aiven-kafka-connect-jdbc/releases/tag/v6.6.0 — OneCricketeer, Aug 18 '21 at 12:56
downloaded file is not available as jar, so i build using gradlew and placed the generated jars in plugin folder — vishnupriya, Aug 18 '21 at 13:26
You're saying that the 40 mb tar/zip files there contain no compiled code? My point is that you'd extract one of those into your plugin path — OneCricketeer, Aug 18 '21 at 13:31
Sorry,I was saying about the sourcecode.Now I got your point,now i downloaded the zip,extract the plugin and added in the same path.Now also issue is not resolved — vishnupriya, Aug 18 '21 at 13:39
If you look at the logs that start with `connect-standalone.sh`, you should see `Added plugin 'io.aiven.connect.jdbc.JdbcSourceConnector'`. If not, then your error starts there — OneCricketeer, Aug 18 '21 at 14:14
@OneCricketeer you are right,I didn't find any logs for `added plugin 'io.aiven.connect.jdbc.JdbcSourceConnector'` .I can see the below logs,after that there is no log for adding plugins. `INFO Loading plugin from: /Kafka/kafka_2.13-2.8.0/Plugins/aiven-kafka-connect-jdbc-6.6.0 (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader:246) INFO Registered loader: PluginClassLoader{pluginLocation=file:/Kafka/kafka_2.13-2.8.0/Plugins/aiven-kafka-connect-jdbc-6.6.0/} (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader:269)` — vishnupriya, Aug 18 '21 at 15:19

score 0 · Answer 1 · answered Aug 18 '21 at 16:11

Not able to reproduce...

Maybe one of these steps will help

$ cd ~/workspace
$ mkdir aiven-kafka-connect-jdbc
$ curl -kL https://github.com/aiven/aiven-kafka-connect-jdbc/releases/download/v6.6.0/aiven-kafka-connect-jdbc-6.6.0.tar | tar -xv -C ./aiven-kafka-connect-jdbc

$ grep -e '^plugin.path=' /usr/local/etc/kafka/connect-standalone.properties
plugin.path=/home/me/workspace/aiven-kafka-connect-jdbc

$ connect-standalone /usr/local/etc/kafka/connect-standalone.properties ~/workspace/sqlite-standalone.properties
...
[2021-08-18 11:07:24,260] INFO Registered loader: PluginClassLoader{pluginLocation=file:/home/me/workspace/aiven-kafka-connect-jdbc/aiven-kafka-connect-jdbc-6.6.0/} (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader:269)
[2021-08-18 11:07:24,260] INFO Added plugin 'io.aiven.connect.jdbc.JdbcSourceConnector' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader:198)
[2021-08-18 11:07:24,261] INFO Added plugin 'io.aiven.connect.jdbc.JdbcSinkConnector' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader:198)
... 
[2021-08-18 11:07:26,276] INFO Kafka Connect started (org.apache.kafka.connect.runtime.Connect:57)
[2021-08-18 11:07:26,296] INFO JdbcConfig values:
    batch.max.rows = 100
    catalog.pattern = null
    connection.attempts = 3
    connection.backoff.ms = 10000
    connection.password = null
    connection.url = jdbc:sqlite:/home/me/workspace/test.db

Someother plugin are getting loaded other than Source and sink connectors, — vishnupriya, Aug 19 '21 at 05:13
Sure, Kafka comes with a few default plugins. I only extracted the relevant logs — OneCricketeer, Aug 19 '21 at 05:15
`Added plugin 'org.apache.kafka.connect.connector.policy.AllConnectorClientConfigOverridePolicy' Added plugin 'org.apache.kafka.connect.connector.policy.PrincipalConnectorClientConfigOverridePolicy' 'org.apache.kafka.connect.connector.policy.NoneConnectorClientConfigOverridePolicy' ` — vishnupriya, Aug 19 '21 at 05:15
Not sure what to tell you. I tested using the latest Kafka Connect on both Mac and Docker — OneCricketeer, Aug 19 '21 at 05:19

score 0 · Answer 2 · answered Oct 05 '21 at 20:24

Make sure you are running on Java 11 or higher.

As stated at the top in their README, The Aiven GCS Sink Connector requires Java 11+, and your Java version is likely older than that.

You can check your current JVM by running java --version.

To download the latest version of Java visit here.

kafka connect not loading manually built connectors

2 Answers2