Stack Overflow
Stack Overflow

Questions tagged [debezium]

Debezium is an open source distributed platform to stream database changes. Use this tag for questions related to using, configuring or troubleshooting Debezium.

Overview

From [https://debezium.io/]:

Debezium is an open source distributed platform for change data capture.

Debezium uses Kafka to record all change events. It supports connectors for:

  • MySQL
  • Oracle
  • MongoDB
  • PostgreSQL
  • SQL Server (colloquially "MSSQL" or "MS SQL")

References

Related Tags

1029 questions
48
votes
3 answers

Kafka Connect JDBC vs Debezium CDC

What are the differences between JDBC Connector and Debezium SQL Server CDC Connector (or any other relational database connector) and when should I choose one over another, searching for a solution to sync between two relational databases? Not…
Ofek Hod
  • 3,544
  • 2
  • 15
  • 26
22
votes
1 answer

debezium could not access file "decoderbufs" using postgres 11 with default plugin pgoutput

I'm new to kafka, I'm trying to use the debezium postgres connector. but even using postgres version 11 with the standard plugin I get this error: org.apache.kafka.connect.errors.ConnectException: org.postgresql.util.PSQLException: ERROR: could not…
Davi Resio
  • 653
  • 2
  • 11
  • 21
15
votes
4 answers

Is Event sourcing using Database CDC considered good architecture?

When we talk about sourcing events, we have a simple dual write architecture where we can write to database and then write the events to a queue like Kafka. Other downstream systems can read those events and act on/use them accordingly. But the…
RBanerjee
  • 957
  • 1
  • 9
  • 18
12
votes
1 answer

Kafka Producer cannot validate record wihout PK and return InvalidRecordException

I have error with my kafka producer. I use Debezium Kafka connectors V1.1.0 Final and Kafka 2.4.1 . For tables with pk, all tables are flushed clearly, but unfortunately for tables with no pk on it, it give me this error: [2020-04-14 10:00:00,096]…
YVS1997
  • 682
  • 1
  • 7
  • 19
11
votes
1 answer

Debezium with AWS MSK NOT_ENOUGH_REPLICAS

I have a running debezium cluster in AWS, no issues with that. I want to give a try with AWS MSK. So I launched a cluster. Then I launched an EC2 for running my connectors. Then installed confluent-kafka sudo apt-get update && sudo apt-get install…
TheDataGuy
  • 2,712
  • 6
  • 37
  • 89
11
votes
1 answer

Problem executing goal wildfly swarm plugin, API incompatibility, java.lang.AbstractMethodError

I am trying to build the debezium-swarm-demo app in the debezium-examples github repository The build does not finish and gives me Execution default of goal org.wildfly.swarm:wildfly-swarm-plugin:2018.5.0:package failed: An API incompatibility was…
anushka
  • 345
  • 4
  • 14
10
votes
3 answers

Updating a Debezium MySQL connector with table whitelist option

I'm using the Debezium (0.7.5) MySQL connector and I'm trying to understand what is the best approach if I want to update this configuration with the option table.whitelist. Let's say I create a connector, something like this: curl -i -X POST -H…
japoneizo
  • 508
  • 1
  • 5
  • 15
10
votes
1 answer

replication slot already exists

Whenever I restart the debezium kafka-connect container, or deploy another instance, I get the following error: io.debezium.jdbc.JdbcConnectionException: ERROR: replication slot "debezium" already exists at…
Zach Mays
  • 101
  • 1
  • 1
  • 6
10
votes
3 answers

Unable to convert Kafka topic data into structured JSON with Confluent Elasticsearch sink connector

I'm building a data pipeline using Kafka. Data flow is as follows: capture data change in mongodb and have it sent to elasticsearch. MongoDB version 3.6 shard cluster Kafka Confuent Platform 4.1.0 mongoDB source connector : debezium…
higee
  • 402
  • 1
  • 7
  • 16
8
votes
1 answer

Kafka Connect | Cannot complete request because of a conflicting operation

1) We have 3 node kafka & kafka connect cluster 2) We are running kafka-connect on kafka nodes only in distributed mode 3) When i am trying to create a connector using below configuration : { "name": "connector-state-0", "config":…
Sahil Gupta
  • 121
  • 1
  • 7
7
votes
0 answers

Debezium postgres incremental snapshot performance issues

I am trying to use debezium incremental snapshots in the latest debezium (1.7) and postgres (V13). For testing, I populated a table with 1M rows, each row is 4KB with a UUID primary key and 20 varchar columns. Since I just wanted to measure snapshot…
sha
  • 614
  • 6
  • 16
7
votes
1 answer

Change-data-capture from Postgres SQL to kafka topics using standalone mode Kafka-connect

I have been trying to get data from postgres sql to kafka topics using the following command /bin connect-standalone.properties config/connect-standalone.properties postgres.sproperties, but am facing several issues with it here are the contents of…
Snehil Singh
  • 207
  • 3
  • 10
7
votes
2 answers

kafka s3 sink connector crashed when It gets NULL data

I had a working s3 sink connector until the source connector sent a NULL value; s3 connector crashed. The problem occured when I deleted a record from MS SQL db. The source connector shipped the deletion information to s3 connector and s3 connector…
Tireli Efe
  • 160
  • 2
  • 11
7
votes
2 answers

Kafka Connect date handling of debezium generated events

I'm using debezium SQL Server to track changes on a production base. The topic is created, CDC is working like a charm, but when trying to use jdbcSinkConnector to dump the data in another Sql Server DB, I'm encountering the following…
nicolasL
  • 136
  • 1
  • 7
7
votes
3 answers

Are there logical decoding plug-ins available for Cloud SQL PostgreSQL?

I'm facing the following situation: I have to do a logical replication of my Cloud SQL PostgreSQL instance to an external PostgreSQL database and vice-versa. More specifically, the CloudSQL and the external instance will contain some master and…
d4nielfr4nco
  • 635
  • 1
  • 6
  • 17
1
2 3
…
68 69