12

How do I use Kafka Connect adapters with Amazon MSK? As per the AWS documentation, it supports Kafka connect but not documented about how to setup adapters and use it.

fmcmac
  • 140
  • 1
  • 9
Mrigank Kishore Varma
  • 376
  • 1
  • 3
  • 12

3 Answers3

29

Edit Oct 2021: MSK Connect has been launched, see https://aws.amazon.com/blogs/aws/introducing-amazon-msk-connect-stream-data-to-and-from-your-apache-kafka-clusters-using-managed-connectors/

AFAIK Amazon MSK does not provide managed connectors, so you have to run them yourself. This is done by running the Kafka Connect worker process (a JVM) and then providing it one or more connector configurations to run.

From the point of view of a Kafka Connect worker it just needs a Kafka cluster to connect to; it shouldn't matter whether it's MSK or on-premises, since it's ultimately 'just' a consumer/producer underneath.

You can see more, including a live demo, here: https://rmoff.dev/bbuzz19-kafka-connect

For an example of configuring Kafka Connect to use a cloud-hosted Kafka platform (in this case, Confluent Cloud), see this article.

If you are interested in managed connectors in the Cloud, check out the connectors that are provided in Confluent Cloud.

Disclaimer: I work for Confluent :)

Robin Moffatt
  • 30,382
  • 3
  • 65
  • 92
  • Am trying to connect to MongoDB atlas within VPC from MSK connect and it throwing a timeout error. Do you know how to make it work? – Jerin A Mathews Nov 05 '21 at 08:36
4

AWS now supports MSK Connect, a new feature of MSK service based on Kafka Connect allowing you to deploy managed Kafka connectors built for Kafka connect

Check the announcement here: https://aws.amazon.com/blogs/aws/introducing-amazon-msk-connect-stream-data-to-and-from-your-apache-kafka-clusters-using-managed-connectors/

Shainberg
  • 96
  • 1
  • 4
  • Am trying to connect to MongoDB atlas within VPC from MSK connect and it throwing a timeout error. Do you know how to make it work? – Jerin A Mathews Nov 05 '21 at 08:36
  • Please share more information about your architecture.. where is you MSK reside? private subnets? public subnet? if it is in private - does it have a NAT gateway? Is your MongoDB exposed to the world? Do you have a VPC peering between your and VPC and MongoDB atlas's VPC? What about your Security group configuration? ACL? – Shainberg Nov 06 '21 at 14:16
  • I am using MSK within VPC( the mongodb have vpc peering to this VPC), with private subnets that have natgateways. My backend server is using the same VPC and subnets to connect to the mongodb. I have given more details in this post-https://stackoverflow.com/questions/69849744/connect-to-mongodb-atlas-within-aws-vpc-using-aws-msk-connector When I open the MongoDB to the world by whitelisting 0.0.0.0 IP, the MSK connector is getting connected to the DB, and the MSK is getting data. But when i remove that IP whitelisting then it's not working. – Jerin A Mathews Nov 07 '21 at 17:42
  • I tried everything that I can think of, and it's still returned an error (Exception opening socket). I think it's an issue with the MSK connect since it's just released on sept2021. Am planning to run the MongoDB Kafka connector in an EC2 and try – Jerin A Mathews Nov 07 '21 at 17:45
  • Are you sure your traffic is going trough the internal network (using the VPC Perring) and not using the outside world? That's might explain the fact that after changing the whitelisting you get the traffic. Check your DNS configuration in VPC peering and verify MongoDB DNS is pointed to an private IP address – Shainberg Nov 08 '21 at 20:09
  • Am sure it is going through internal network, because am using the same vpc and subnets in my ec2 server(backend code ) to connect to db, and its working fine. I will check everything again and confirm – Jerin A Mathews Nov 11 '21 at 06:04
  • @JerinAMathews did you figure this out in the end? – Martin W Jul 27 '22 at 14:37
2

There are two aspects to this

  1. Kafka Connect is a framework which should be deployed separately from kafka brokers. MSK only provides kafka brokers. If you want to use Kafka Connect with MSK you would need to use EC2 instances and deploy the kafka binaries.Kafka Connect framework is bundled along with kafka

  2. Coming to connectors if you donot have a confluent subscription or similar - I am afraid your choices get very limited. But having said you can always write your own connectors. Writing new connectors is not that difficult rather you can apply your business specific logic and be on your way quite quickly.

floating_hammer
  • 409
  • 3
  • 10