0

I have a managed kafka cluster running in the cloud on Aiven (GCP). I'm trying to use the Debezium SQL Server connector to stream CDC records from our on-premise database.

Does anybody know how to connect from kafka in the cloud to an on-premise database? Can this be done through some sort of VPN connection or do I need to configure a VPC through google cloud? Appreciate any advice.

Stennie
  • 63,885
  • 14
  • 149
  • 175
ThinkBig
  • 11

1 Answers1

0

You can use Debezium to stream CDC records from databases to a Kafka cluster, yes.

Here's an example with a different managed Kafka service, Confluent Cloud.

The main thing is getting your security settings right, as well as specifying the appropriate database.history.consumer and database.history.producer configuration.

Robin Moffatt
  • 30,382
  • 3
  • 65
  • 92
  • Thanks for the details. In the example it looks as though Kafka Connect/Debezium is installed within the local network and pushing data to Confluent in the cloud. Aiven provides a managed version of Kafka connect which is also cloud based. Any idea in this scenario how to connect to an on-prem database? – ThinkBig Nov 11 '19 at 11:52
  • That's probably a question best asked of Aiven themselves. – Robin Moffatt Nov 11 '19 at 12:48
  • Reponse from Aiven tech support: 1. It is not possible to setup a VPN from the VPC where Aiven services are running 2. Fundamentally, the VPC where Aiven services are deployed in has to be able to reach/connect to the on-prem MS SQL server. Therefore, you will have to open ports in your firewall to anable this connectivity. 3. Transit gateway may be an option. In this approach, the Aiven services' VPC and your VPN would all connect to the transit gateway, and the gateway would know how to route traffic. Transit gateway feature is something that we currently have under development. – ThinkBig Nov 13 '19 at 11:45