Questions tagged [confluent-kafka-go]
35 questions
6
votes
0 answers
Confluent Kafka configure initial connection retry timeout
I have a go project with confluent Kafka and I want to change it's initial connection timeout. If the kafka is not available on the first try I want to retry connecting to it every few seconds and give up retrying after 10 min.
By default it is…
Rob Minasyan
- 390
- 2
- 14
3
votes
0 answers
Unable to debug Go code on Windows using VSCode/GoLand (Getting '%1 is not a valid Win32 application')
When I am trying to debug this Go application using VS Code and dlv, I get the following error:
Failed to launch: could not launch process: fork/exec
C:\Users\MyUser\Workspaces\goworkspace\github.com...__debug_bin.exe:%1
is not a valid Win32…
SDB
- 61
- 1
- 4
2
votes
0 answers
Confluent Cloud Kafka Go v2.1.1 : ssl/certs and unregistered scheme error
We have recently upgraded from v1 to v2 of the library. We are encountering the below errors in our bootstrap logs:
sasl_ssl://xxx-xxx-xxx-xxx.yyy.zzz.aaa.confluent.cloud:9092/bootstrap: error:80000002:system library::No such file or directory:…
i.am.it
- 21
- 3
2
votes
1 answer
Does Confluent kafka go package compatible with ubuntu 22.04?
I was using confluent kafka package for my data streaming server in golang using ubuntu 20.04 and now I changed my os to ubuntu 22.04.
Now im getting these errors:
- kafka.producer/producer.go:18:26: undefined: kafka.Producer
-…
Suriya Varman N
- 79
- 5
2
votes
1 answer
How can I fix my docker build failing while my go build succeeds? Dockerfile includes go mod download
I am using "github.com/confluentinc/confluent-kafka-go/kafka" in a very simple consumer. It's pretty much what confluent has as a kafka/go tutorial.
"go build ." and "go run ." succeed, "docker build ." does not.
The error is:
> [builder 7/7] RUN…
Andy Troschke
- 441
- 1
- 5
- 13
2
votes
1 answer
Resolving InvalidTimestampException with Confluent-Kafka-Go v1.8.x and Kafka v3.2.x: Seeking Clarification on Timestamp and Leader Epoch Issues
Understanding and Troubleshooting Timestamp-related Issues in Confluent-Kafka-Go and Kafka Server Integration...
I'd like to ask if anyone has encountered the following issue when using confluent-kafka-go v1.8.x to connect to a Kafka v3.2.x server:…
Howard Chen
- 23
- 4
2
votes
0 answers
kafka `ReadMessage` face timeout even though there is message in the queue
Just right after creating kafka.Consumer instance, I want to adjust the offsets with following code, decrease all assigned partitions offset(for specific consumer member) and ensure the previous messages are processed. The offsets are updated after…
Maryam
- 660
- 6
- 19
2
votes
1 answer
Processing csv files with a queue using Kafka to insert to postgres database and return status message
I looking for some best practices/advice to handle processing CSV file for inserting into the database with a queue mechanism (Kafka)
So here what i will do :
Create a new SQL table Service Request to store information of the user request like…
Faris Dewantoro
- 1,597
- 4
- 17
- 31
1
vote
0 answers
Kafka golang consumers do not rebalance as expected
I'm trying to test rebalancing consumers. I've read about consumer groups and they don't work as I expected. I have a two consumers running in the same group like so:
package main
import (
"errors"
"net"
"os"
"time"
…
Omri. B
- 375
- 1
- 13
1
vote
1 answer
How to add Replication factor to a topic?
conf := &kafka.ConfigMap{
"bootstrap.servers": "127.0.0.1:9092, 127.0.0.1:9093, 127.0.0.1:9094",
"min.insync.replicas":3,
}
producer, err := kafka.NewProducer(conf)
if err != nil {
log.Println("Error initializing…
Suriya Varman N
- 79
- 5
1
vote
0 answers
confluent-kafka-go consumer batch commits
I prefer confluent-kafka-go driver. It's fast but it needs some workaround. The most tricky part is commiting messages. For the better performance we shouldn't commit every message, it's better to do commit periodically. But to make this approach…
Yura
- 969
- 14
- 33
1
vote
0 answers
Kafka producer failing with SIGSEGV panic when closing the producer connection
Application Functionality : Consumes messages from kafka queue and puts in buffered channel . From buffered channel, go function reads from that channel and do required message processing. if this message processing fails , it will publish back to…
Vani Polnedi
- 595
- 2
- 4
- 19
1
vote
0 answers
Kafka Go Client using Go Modules
I'm trying to implement Kafka producer in Go using Confluent-Kafka Client for Go, following the documentation here https://github.com/confluentinc/confluent-kafka-go#using-go-modules
My setup: Have a Lambda Function that reads Kafka Messages from…
flowAlong
- 77
- 10
1
vote
1 answer
Kafka persistent storage is growing, though cleanup and retention policies are set
We are seeing a steady growth in the persistent storage for Kafka, around 10% per day - effectively implying the configs are not tuned. There are 3 brokers and 5 topics. Retention was configured for 2 hours.
Following are the config parameters set…
prof
- 21
- 3
1
vote
1 answer
kafka consumer with confluent-kafka-go change offset
I create a new consumer with this configuration:
c, err := kafka.NewConsumer(&kafka.ConfigMap{
"bootstrap.servers": addresses,
"group.id": "my_group",
"auto.offset.reset": "earliest",
…
Maryam
- 660
- 6
- 19