i have found a lot of responses to this question, but none works for me.. the error from the kafka-connects log is this:
Caused by: java.sql.SQLException: No suitable driver found for jdbc:postgresql://postgres-dest:5432/shipment_db?user=postgres&password=postgres
This is the compose where kafka connect is runned:
version: '3.3'
services:
postgres:
container_name: postgres
ports:
- '5432:5432'
environment:
- POSTGRES_USER=postgres
- POSTGRES_PASSWORD=postgres
- POSTGRES_DB=shipment_db
- PGPASSWORD=password
image: 'debezium/postgres:13'
postgres-dest:
container_name: postgres-dest
ports:
- '5433:5432'
environment:
- POSTGRES_USER=postgres
- POSTGRES_PASSWORD=postgres
- POSTGRES_DB=shipment_db
- PGPASSWORD=password
image: 'debezium/postgres:13'
zookeeper:
container_name: zookeeper
ports:
- '2181:2181'
- '2888:2888'
- '3888:3888'
image: 'debezium/zookeeper:1.7'
kafka:
container_name: kafka
ports:
- '9092:9092'
links:
- 'zookeeper:zookeeper'
image: 'debezium/kafka:1.7'
environment:
ZOOKEEPER_CONNECT: zookeeper:2181
connect:
image: debezium/connect:1.7
hostname: connect
container_name: connect
ports:
- 8083:8083
environment:
BOOTSTRAP_SERVERS: kafka:9092
GROUP_ID: 1
CONFIG_STORAGE_TOPIC: my_connect_configs
OFFSET_STORAGE_TOPIC: my_connect_offsets
STATUS_STORAGE_TOPIC: my_connect_statuses
CONNECT_BOOTSTRAP_SERVERS: kafka:9092
CONNECT_GROUP_ID: connect-cluster-A
CONNECT_PLUGIN_PATH: /kafka/connect,/kafka/data
#EXTERNAL_LIBS_DIR: /kafka/external_libs,/kafka/data
#CLASSPATH: /kafka/connect
#KAFKA_CONNECT_PLUGINS_DIR: /kafka/connect, /kafka/data
volumes:
- type: bind
source: ./plugins
target: /kafka/data
depends_on:
- zookeeper
- kafka
- postgres
links:
- zookeeper
- kafka
- postgres
- postgres-dest
the CONNECT_PLUGIN_PATH (plugin.dir) is setted to /kafka/data and into are inserted the plugin for jdbc with the postgresql driver in the same folder.
When i load the kafka-connect's container i found in the logs:
2021-12-23 15:44:43,524 INFO || Loading plugin from: /kafka/data/postgresql-42.2.19.jar
then i suppose that driver is loaded correctly. The connector is downloaded from here jdbc connector
The configuration of the connector is the follow:
curl -H 'Content-Type: application/json' --data '
{
"name": "jdbc-sink",
"config": {
"connector.class": "io.confluent.connect.jdbc.JdbcSinkConnector",
"tasks.max": "1",
"topics": "accounts",
"connection.url": "jdbc:postgresql://postgres-dest:5432/shipment_db?user=postgres&password=postgres",
"auto.create": "true",
"insert.mode": "upsert",
"pk.fields": "id",
"pk.mode": "record_value"
}
}
}' http://localhost:8083/connectors
I have tryed many solution:
- setup the CLASSPATH with /kafka/data ( the driver + plugins dir )
- setup the environment variable KAFKA_CONNECT_PLUGINS_DIR to "/kafka/connect, /kafka/data"
but none of this solutions works.. I think that the problem are in the connection between kafka-connect and postgres-dest container but i don't undestand where the problem is, given that the connection-url of connector seems writed correctly.
Thanks in advance for any response.