Stack Overflow
Stack Overflow

Questions tagged [aws-msk]

For questions specific to the usage and features of Amazon Managed Streaming for Apache Kafka. Use this tag in combination with the more generic [apache-kafka] tag when appropriate.

371 questions
40
votes
1 answer

Amazon Managed Streaming for Kafka- MSK features and performance

I'm evaluating AWS Managed Service Kafka (MSK) and I know that currently, it is in preview mode, hence might not have all features or proper documentation. I tried setting up the msk cluster and was validating whether msk could fulfill all the use…
britney
  • 511
  • 1
  • 4
  • 4
14
votes
2 answers

kafka-python raise UnrecognizedBrokerVersion Error

I am getting this error when constructing KafkaProducer with the kafka-python package: [ERROR] UnrecognizedBrokerVersion: UnrecognizedBrokerVersion Traceback (most recent call last): File "/var/lang/lib/python3.7/imp.py", line 234, in load_module …
Rene B.
  • 6,557
  • 7
  • 46
  • 72
12
votes
1 answer

Amazon Kinesis vs AWS Manage Service Kafka (MSK) - (Connect from on-prem)

I'm evaluating AWS Kinesis vs Managed Service Kafka (MSK). Our requirement is sending some messages (JSON) to AWS to from the on-prem system (system develop using c++). Then we need to persist above messages into the relational database like…
HASH
  • 123
  • 1
  • 5
10
votes
2 answers

How AWS MSK and Confluent Schema Registry and Confluent Kafka connect recommended to use together?

We are planning to use AWS MSK service for Managed Kafka and Schema Registry and Kafka Connect services from Confluent together to run our connectors (Elasticsearch Sink Connector). We have planned to run Schema Registry and Connectors in EC2. As…
Suresh Sanghani
  • 131
  • 1
  • 8
10
votes
2 answers

How to set auto.create.topics.enable as default config on AWS MSK cluster

Our AWS MSK cluster is not automatically creating topics when a producer first publishes a message to the topic. This is surprising because the default behaviour according to https://kafka.apache.org/documentation/#brokerconfigs should have…
rpassza
  • 226
  • 1
  • 2
  • 8
9
votes
2 answers

AWS Kafka (MSK) - How to generate Keystore and truststore and use the same in my Spring Cloud Stream application?

Is there any info as to how can I use the AWS MSK details in my Spring Cloud Stream application ? I believe we need to generate a keystore and truststore and then incorporate the same in our application ? I went through the "Client Authentication"…
Kenny Weeler
  • 153
  • 1
  • 3
  • 15
9
votes
1 answer

AWS MSK - Timeout when creating Kafka topic with ACL turned-on

I'm using AWS MSK and I want to enable ACLs but I'm unable to create a topic when ACLs are turned-on. I'm using the command-line tools for all the operations. Here's a summary of what I'm doing: Create a fresh cluster Create a topic - this works…
Araf
  • 353
  • 5
  • 11
7
votes
1 answer

Connect Python to MSK with IAM role-based authentication

I've written a python script with aiokafka to produce and consume from a Kafka cluster in AWS MSK, I'm running the script from a EC2 instance that is in the same VPC as my cluster and when I try to connect my script to a cluster it refuse to accept…
Israel Calderón
  • 81
  • 1
  • 4
7
votes
2 answers

How to get brokers endpoint of Amazon MSK as an output

we have an AWS cloudformation template, through which we are creating the Amazon MSK(Kafka) cluster. which is working fine. Now we have multiple applications in our product stack which consume the Brokers endpoints which is created by the Amazon…
chitender kumar
  • 394
  • 4
  • 21
6
votes
1 answer

Why is AWS MSK Kafka broker constantly disconnecting and reconnecting the consumer group

I have AWS MSK Kafka cluster with 2 brokers. From the logs I can see (on each broker) that they are constantly rebalancing. Every minute I can see in logs: Preparing to rebalance group amazon.msk.canary.group.broker-1 in state PreparingRebalance…
amorfis
  • 15,390
  • 15
  • 77
  • 125
6
votes
1 answer

KSQL in AWS MSK

Does anyone have any idea if AWS MSK (Managed Streaming for Kafka) supports KSQL? I have seen many videos and documentations for KSQL on confluent platform but not for AWS MSK. Please let me know if it supports and also let me know if there is any…
Bala
  • 99
  • 2
  • 6
5
votes
0 answers

Unable to connect to kafka MSK using segmentio/kafka-go

I am trying to create a function that checks if a topic exists in my AWS MSK func KafkaTopicExist(kafkabroker string, topic string) bool { conn, err := kafka.Dial("tcp", kafkabroker) if err != nil { log.WithError(err).Warn("Kafka broker…
MBA
  • 353
  • 2
  • 11
5
votes
1 answer

Using AWS Glue Schema Registry for Node.js or Ruby on Rails

Looking at the documentation for AWS Glue Schema Registry it seems like you can't use it for Node.js or Ruby on Rails: The AWS Glue Schema Registry supports AVRO (v1.10.2) data format and Java language support, with other data formats and languages…
Sandip Subedi
  • 1,039
  • 1
  • 14
  • 34
5
votes
1 answer

MSK Not Deleting Old Messages

I have three MSK clusters; dev, nonprod & prod. They all have the following cluster configuration - there is no topic level…
Garreth
  • 1,057
  • 2
  • 9
  • 24
5
votes
1 answer

Problems with Amazon MSK default configuration and publishing with transactions

Recently we have started doing some testing of our Kafka connectors to MSK, Amazon's managed Kafka service. Publishing records seem to work fine however not when transactions are enabled. Our cluster consists of 2 brokers (because we have 2 zones)…
Gray
  • 115,027
  • 24
  • 293
  • 354
1
2 3
…
24 25