debezium could not access file "decoderbufs" using postgres 11 with default plugin pgoutput

Question

I'm new to kafka, I'm trying to use the debezium postgres connector. but even using postgres version 11 with the standard plugin I get this error: org.apache.kafka.connect.errors.ConnectException: org.postgresql.util.PSQLException: ERROR: could not access file "decoderbufs": No such file or directory

To run kafka / debezium I'm using the image of the fast-data-dev docker as you can see below

  # this is our kafka cluster.
  kafka-cluster:
    image: landoop/fast-data-dev:latest
    environment:
      ADV_HOST: 127.0.0.1         # Change to 192.168.99.100 if using Docker Toolbox
      RUNTESTS: 0                 # Disable Running tests so the cluster starts faster
    ports:
      - 2181:2181                 # Zookeeper
      - 3030:3030                 # Landoop UI
      - 8081-8083:8081-8083       # REST Proxy, Schema Registry, Kafka Connect ports
      - 9581-9585:9581-9585       # JMX Ports
      - 9092:9092                 # Kafka Broker

after running i can open my localhost: 3030 to choose the debezium connector, i configured it this way:

and I'm using aws postgres rds in version 11.5 I saw several tutorials using wal2json, but I didn't find it in rds.extensions and didn't see a way to add it. Anyway, as of version 10, debezium can use pgoutput and apparently no configuration is necessary.

the rds.logical_replication property is set to 1 when executing SHOW wal_level; in the terminal I see that it returns logical in the documentation says that you have to set max_wal_senders = 1 and max_replication_slots = 1 put in the rds the minimum is 5, so I left the default that is 10

I did not define the role REPLICATION because from what I understand in the rds there is no way

in this image you can see the version used is 11.5

but I get the error as you can see below

I found the problem, there was an option that I clicked and showed all the values, many defaults that were already filled, the plugin.name was with decoderbufs, it was just switching to pqoutput that worked — Davi Resio, Jan 30 '20 at 10:57
@ Vikrant Goel the problem was this, i was not finding this option. I found and put pgoutput. thanks — Davi Resio, Feb 27 '20 at 01:22

score 49 · Accepted Answer · edited Oct 31 '20 at 08:26

You haven't set "plugin.name" property to "pgoutput" in your Debezium connector properties which you've already figured out. But this answer is for others who don't know where to set this option and for more clarity.

In the following connector properties:

As you haven't set plugin.name option it is taking the default value which is decoderbufs and that's why you were getting the following error:

Setting the "plugin.name" explicitly to "pgoutput" should solve the issue.

{ 
  "connector.class": "io.debezium.connector.postgresql.PostgresConnector", 
  "database.user": "postgres",
  "database.dbname": "xxxxx",
  "tasks.max": "1", 
  "database.hostname": "xxxx.rds.amazonaws.com", 
  "database.password": "xxxx", 
  "database.server.name": "database-1",
  "database.port": "5432",
  "plugin.name": "pgoutput" --> this property
}

debezium could not access file "decoderbufs" using postgres 11 with default plugin pgoutput

1 Answers1

Linked