Stack Overflow
Stack Overflow

Questions tagged [amazon-msk]

27 questions
12
votes
3 answers

Kafka Connect with Amazon MSK

How do I use Kafka Connect adapters with Amazon MSK? As per the AWS documentation, it supports Kafka connect but not documented about how to setup adapters and use it.
Mrigank Kishore Varma
  • 376
  • 1
  • 3
  • 12
11
votes
1 answer

Debezium with AWS MSK NOT_ENOUGH_REPLICAS

I have a running debezium cluster in AWS, no issues with that. I want to give a try with AWS MSK. So I launched a cluster. Then I launched an EC2 for running my connectors. Then installed confluent-kafka sudo apt-get update && sudo apt-get install…
TheDataGuy
  • 2,712
  • 6
  • 37
  • 89
8
votes
2 answers

Can talk to Zookeeper but not to the message brokers

I'm using kafka-python to produce messages for a Kafka 2.2.1 cluster (a managed cluster instance from AWS's MSK service). I'm able to retrieve the bootstrap servers and establish a network connection to them, but no message ever gets through.…
Kilian Foth
  • 13,904
  • 5
  • 39
  • 57
4
votes
4 answers

Access to amazon msk cluster locally

I have an amazon msk cluster (kafka) in a vpc and I can access it through a client (EC2 instance) that is in the same vpc. my question is .. can i access my cluster from a client that is on my local computer? How can I do this configuration? I tried…
CARLV
  • 71
  • 1
  • 4
3
votes
4 answers

AWS MSK lambda concurrent consumers

AWS Lambda Supports Parallelization Factor for Kinesis and DynamoDB Event Sources. But its not supported for MSK. Can we create a reserved concurrency of the Lambda function and would it help to concurrently consume from MSK topic
dvlpr
  • 311
  • 3
  • 17
2
votes
1 answer

kafka connect mongo on kafka MSK

I am using Kafka MSK in AWS. So we don't have native kafka connect with all required connectors like on confluent. Actually I work with kakfa mongo connector and I want to find a way to push the kafka mongo connector jar to an on an instance of…
scalacode
  • 1,096
  • 1
  • 16
  • 38
1
vote
1 answer

Meaning of SumOffsetLag in MsK cloudwatch data?

We have kafka which we are monitoring using grafana dashboad provided by aws cloud watch. From the past few days we are seeing some abnormality. I dont have Idea much about SumOffsetLag metric parameter. How this spiked value can degrade the…
dead programmer
  • 4,223
  • 9
  • 46
  • 77
1
vote
1 answer

How to check the undelivered messages in Amazon MSK?

I was going through AWS 's service MSK which is Managed Service for Kafka. I wanted to check how many messages have not been consumed by the consumer after I send them from a producer. Is the only way to do it via enabling the Monitoring options AWS…
sameerawsnoob
  • 105
  • 1
  • 10
1
vote
1 answer

AWS lambda from MSK cluster missing kafka message header

I want to use AWS lambda to consumer kafka messages, but the events I received don't contain kafka headers, that are very critical. Is there a way to get messages with headers info?
yiles2003
  • 11
  • 1
1
vote
2 answers

How To Run Kafka Camel Connectors On Amazon MSK

Context: I followed this link on setting up AWS MSK and testing a producer and consumer and it is setup and working correctly. I am able to send and receive messages via 2 separate EC2 instances that both use the same Kafka cluster (My MSK cluster).…
Mike R
  • 464
  • 5
  • 16
1
vote
1 answer

AWS Lambda Event Payload for MSK trigger

I'm trying to deserialize value to restore Kafka message value. But i can not find proper deserializer. I receive following…
Tie Kapch
  • 31
  • 5
1
vote
0 answers

Issue with AWS MSK with TLS, with Spring-kafka

Anyone used AWS MSK with TLS, with Spring-kafka, below are the details of our application, with the config below, the application is not working we have deployed our consumers/producers on Fargate kafka-client version is 2.1.1 below are the…
DEE
  • 373
  • 5
  • 17
1
vote
1 answer

Sarama unable to produce message for Amazon MSK version 2.3.1

I was using sarama golang library for pushing the messages to Amazon MSK. Till now I was using msk version 2.2.1 my code was working fine, But now the msk version has been changed to 2.3.1. Now, I am unable to push the message to the…
Chetan Kumar
  • 1,331
  • 1
  • 10
  • 16
0
votes
1 answer

AWS MSK(kafka) producer transaction commit Timeout

I have been trying to send messages to AWS MSK using python confluent-kafka library. I want to ensure single delivery of each message which is why I am using transaction-based producer. I am currently sending 500k messages per transaction. The…
Puneet Jain
  • 1
  • 1
0
votes
1 answer

Getting timeout while creating ACL for Amazon Kafka

I'm setting up a public kafka cluster with SCRAM auth on AWS MSK. I created the cluster, attached a secret from KMS and made it public. When I'm trying to connect it with kcat, I'm getting topic authorization error. $ echo "message" | kcat -L -b…
krsoni
  • 539
  • 7
  • 11
1
2