1

I observe that row updates are streamed slowly when I stop and restart the Debezium connector. It sometimes takes almost nine minutes for any row updates to stream instantaneously. During this period, row updates start off by taking over a minute. Gradually the time decreases to being instantaneous. This "waiting period" is sometimes less.

I'm using the official Debezium Docker image. My database is PostgreSQL 15.

Here's how I create the connector:

curl -H 'Content-Type: application/json' 127.0.0.1:9083/connectors --data '
{
  "name": "my-name",  
  "config": {
    "connector.class": "io.debezium.connector.postgresql.PostgresConnector", 
    "topic.prefix": "prefix",
    "database.hostname": "myhost.net", 
    "database.port": "5432", 
    "database.user": "root", 
    "database.password": "root", 
    "database.server.name": "my_db", 
    "database.dbname" : "my_db", 
    "table.include.list": "my_schema.my_table",
    "plugin.name": "pgoutput",
    "heartbeat.interval.ms": 100
  }
}'

The table in table.include.list is updated infrequently, but I have other tables that are updated frequently, so I had to use heartbeat.interval.ms as per this.

Why are row updates streamed so slowly in this scenario? How can I recover from this scenario?

user2233706
  • 6,148
  • 5
  • 44
  • 86
  • I am running to the same problem with pg13 on a table with 130.000 records. But when I run the same config to another db with around 7000, this doesn't happen. And I encountered this after the snapshot was done. The changes were not coming directly. Did you manage to solve this problem ? – Serkan Apr 20 '23 at 12:20
  • We ended up not using Debezium, but perhaps try adjusting `max_slot_wal_keep_size` (via [this](https://stackoverflow.com/questions/60527214/how-to-limit-wal-size-when-using-postgres-logical-replication-slot)). I think this is related to the WAL, because if you had less records, the streaming happened much faster. – user2233706 Apr 20 '23 at 14:11

0 Answers0