I was trying to connect to kafka broker using Shopify/sarama library. I ran two program, one for producing and one for consuming messages. Both of them fail with the same error message:
[Err]: 2022/11/18 16:59:42 dial tcp: lookup c6dc12e3b671: Temporary failure in name resolution
producer.go
func main() {
logger := log.New(os.Stdout, "[Proudcer]:\t", log.LstdFlags)
sarama.Logger = log.New(os.Stdout, "[Sarama]:\t", log.LstdFlags)
conf := sarama.NewConfig()
conf.ClientID = "producer"
conf.Version = sarama.V3_2_3_0
conf.Producer.Partitioner = sarama.NewRandomPartitioner
conf.Producer.RequiredAcks = sarama.WaitForAll
conf.Producer.Return.Successes = true
prod, err := sarama.NewSyncProducer([]string{"localhost:9092"}, conf)
if err != nil {
log.Fatal(err)
}
message := &sarama.ProducerMessage{
Topic: "message.sent",
Value: sarama.StringEncoder("Message"),
}
for i := 0; i < 5; i += 1 {
_, _, err := prod.SendMessage(message)
if err != nil {
logger.Println(err)
}
time.Sleep(5 * time.Second)
}
}
consumer.go
func main() {
successLogger := log.New(os.Stdout, "[Message consumed]:\t", log.LstdFlags)
errLogger := log.New(os.Stdout, "[Err]:\t", log.LstdFlags)
topic := "message.sent"
sarama.Logger = log.New(os.Stdout, "[Sarama]:\t", log.LstdFlags)
conf := sarama.NewConfig()
conf.ClientID = "consumer"
conf.Version = sarama.V3_2_3_0
consumer, err := sarama.NewConsumer([]string{"localhost:9092"}, conf)
if err != nil {
log.Fatal(err)
}
partitions, err := consumer.Partitions(topic)
if err != nil {
log.Fatal(err)
}
for _, partition := range partitions {
con, err := consumer.ConsumePartition(topic, partition, 0)
if err != nil {
errLogger.Println(err)
}
go func() {
for err := range con.Errors() {
errLogger.Println(err)
}
}()
channel := con.Messages()
for msg := range channel {
var body string
err := json.Unmarshal(msg.Value, &body)
if err != nil {
errLogger.Println(err.Error())
}
successLogger.Println(body)
}
}
}
In producer.go error occures when SendMessage() is called. In consumer.go its ConsumePartition() that gives an error. I ran Kafka with this docker-compose file: https://github.com/bitnami/containers/blob/main/bitnami/kafka/docker-compose.yml
I tried running examples from Github both using Sarama and other libraries and I get the same error (Examples I've tried: https://github.com/ppatierno/kafka-go-examples). I also tried different kafka images with the same result, so I guess there has to be something wrong with my environment. Go version: 1.19.2 OS: Ubuntu 20.04.3 LTS
These are logs program logs:
producer logs
[Sarama]: 2022/11/18 16:38:30 Connected to broker at localhost:9092 (unregistered)
[Sarama]: 2022/11/18 16:38:30 client/brokers registered new broker #1001 at c6dc12e3b671:9092
[Sarama]: 2022/11/18 16:38:30 Successfully initialized new client
[Sarama]: 2022/11/18 16:38:30 producer/broker/1001 starting up
[Sarama]: 2022/11/18 16:38:30 Failed to connect to broker c6dc12e3b671:9092: dial tcp: lookup c6dc12e3b671: Temporary failure in name resolution
[Sarama]: 2022/11/18 16:38:30 Error while sending ApiVersionsRequest to broker c6dc12e3b671:9092: dial tcp: lookup c6dc12e3b671: Temporary failure in name resolution
[Sarama]: 2022/11/18 16:38:30 producer/broker/1001 state change to [open] on message.sent/0
[Sarama]: 2022/11/18 16:38:30 producer/broker/1001 state change to [closing] because dial tcp: lookup c6dc12e3b671: Temporary failure in name resolution
[Sarama]: 2022/11/18 16:38:30 producer/leader/message.sent/0 state change to [retrying-1]
[Sarama]: 2022/11/18 16:38:30 producer/leader/message.sent/0 abandoning broker 1001
[Sarama]: 2022/11/18 16:38:30 producer/broker/1001 input chan closed
[Sarama]: 2022/11/18 16:38:30 producer/broker/1001 shut down
[Sarama]: 2022/11/18 16:38:30 client/metadata fetching metadata for [message.sent] from broker localhost:9092
[Sarama]: 2022/11/18 16:38:30 producer/broker/1001 starting up
[Sarama]: 2022/11/18 16:38:30 producer/broker/1001 state change to [open] on
message.sent/0
[Sarama]: 2022/11/18 16:38:30 producer/leader/message.sent/0 selected broker 1001
[Sarama]: 2022/11/18 16:38:30 producer/leader/message.sent/0 state change to [flushing-1]
[Sarama]: 2022/11/18 16:38:30 producer/leader/message.sent/0 state change to [normal]
[Sarama]: 2022/11/18 16:38:30 Failed to connect to broker c6dc12e3b671:9092: dial tcp: lookup c6dc12e3b671: Temporary failure in name resolution
[Sarama]: 2022/11/18 16:38:30 Error while sending ApiVersionsRequest to broker c6dc12e3b671:9092: dial tcp: lookup c6dc12e3b671: Temporary failure in name resolution
[Sarama]: 2022/11/18 16:38:30 producer/broker/1001 state change to [closing] because dial tcp: lookup c6dc12e3b671: Temporary failure in name resolution
[Sarama]: 2022/11/18 16:38:30 producer/leader/message.sent/0 state change to [retrying-2]
[Sarama]: 2022/11/18 16:38:30 producer/leader/message.sent/0 abandoning broker 1001
[Sarama]: 2022/11/18 16:38:30 producer/broker/1001 input chan closed
[Sarama]: 2022/11/18 16:38:30 producer/broker/1001 shut down
[Sarama]: 2022/11/18 16:38:30 client/metadata fetching metadata for [message.sent] from broker localhost:9092
[Sarama]: 2022/11/18 16:38:30 producer/broker/1001 starting up
[Sarama]: 2022/11/18 16:38:30 producer/broker/1001 state change to [open] on message.sent/0
[Sarama]: 2022/11/18 16:38:30 producer/leader/message.sent/0 selected broker 1001
[Sarama]: 2022/11/18 16:38:30 producer/leader/message.sent/0 state change to
[flushing-2]
[Sarama]: 2022/11/18 16:38:30 producer/leader/message.sent/0 state change to [normal]
[Sarama]: 2022/11/18 16:38:30 Failed to connect to broker c6dc12e3b671:9092: dial tcp: lookup c6dc12e3b671: Temporary failure in name resolution
[Sarama]: 2022/11/18 16:38:30 Error while sending ApiVersionsRequest to broker c6dc12e3b671:9092: dial tcp: lookup c6dc12e3b671: Temporary failure in name resolution
[Sarama]: 2022/11/18 16:38:30 producer/broker/1001 state change to [closing] because dial tcp: lookup c6dc12e3b671: Temporary failure in name resolution
[Sarama]: 2022/11/18 16:38:30 producer/leader/message.sent/0 state change to [retrying-3]
[Sarama]: 2022/11/18 16:38:30 producer/leader/message.sent/0 abandoning broker 1001
[Sarama]: 2022/11/18 16:38:30 producer/broker/1001 input chan closed
[Sarama]: 2022/11/18 16:38:30 producer/broker/1001 shut down
[Sarama]: 2022/11/18 16:38:30 client/metadata fetching metadata for [message.sent] from broker localhost:9092
[Sarama]: 2022/11/18 16:38:30 producer/broker/1001 starting up
[Sarama]: 2022/11/18 16:38:30 producer/broker/1001 state change to [open] on message.sent/0
[Sarama]: 2022/11/18 16:38:30 producer/leader/message.sent/0 selected broker 1001
[Sarama]: 2022/11/18 16:38:30 producer/leader/message.sent/0 state change to [flushing-3]
[Sarama]: 2022/11/18 16:38:30 producer/leader/message.sent/0 state change to [normal]
[Sarama]: 2022/11/18 16:38:30 Failed to connect to broker c6dc12e3b671:9092: dial tcp: lookup c6dc12e3b671: Temporary failure in name resolution
[Sarama]: 2022/11/18 16:38:30 Error while sending ApiVersionsRequest to broker c6dc12e3b671:9092: dial tcp: lookup c6dc12e3b671: Temporary failure in name resolution
[Sarama]: 2022/11/18 16:38:30 producer/broker/1001 state change to [closing] because dial tcp: lookup c6dc12e3b671: Temporary failure in name resolution
[Proudcer]: 2022/11/18 16:38:30 dial tcp: lookup c6dc12e3b671: Temporary failure in name resolution
consumer logs:
[Sarama]: 2022/11/18 16:38:24 Initializing new client
[Sarama]: 2022/11/18 16:38:24 client/metadata fetching metadata for all topics from broker localhost:9092
[Sarama]: 2022/11/18 16:38:24 Connected to broker at localhost:9092 (unregistered)
[Sarama]: 2022/11/18 16:38:24 client/brokers registered new broker #1001 at c6dc12e3b671:9092
[Sarama]: 2022/11/18 16:38:24 Successfully initialized new client
[Sarama]: 2022/11/18 16:38:24 Failed to connect to broker c6dc12e3b671:9092: dial tcp: lookup c6dc12e3b671: Temporary failure in name resolution
[Sarama]: 2022/11/18 16:38:24 Error while sending ApiVersionsRequest to broker c6dc12e3b671:9092: dial tcp: lookup c6dc12e3b671: Temporary failure in name resolution
[Sarama]: 2022/11/18 16:38:24 client/metadata fetching metadata for [message.sent] from broker localhost:9092
[Sarama]: 2022/11/18 16:38:24 Failed to connect to broker c6dc12e3b671:9092: dial tcp: lookup c6dc12e3b671: Temporary failure in name resolution
[Sarama]: 2022/11/18 16:38:24 Error while sending ApiVersionsRequest to broker c6dc12e3b671:9092: dial tcp: lookup c6dc12e3b671: Temporary failure in name resolution
[Err]: 2022/11/18 16:38:24 dial tcp: lookup c6dc12e3b671: Temporary failure in name resolution
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x40 pc=0x6fe132]
goroutine 1 [running]:
main.main()
/home/sebastian/Dokumenty/GO/test/kafka/consumer/main.go:46 +0x472
exit status 2
sebastian@sebastian-OMEN-by-HP-Laptop-15-ce0xx:~/Dokumenty/GO/test/kafka/consumer$ go run main.go
[Sarama]: 2022/11/18 16:59:42 Initializing new client
[Sarama]: 2022/11/18 16:59:42 client/metadata fetching metadata for all topics from broker localhost:9092
[Sarama]: 2022/11/18 16:59:42 Connected to broker at localhost:9092 (unregistered)
[Sarama]: 2022/11/18 16:59:42 client/brokers registered new broker #1001 at c6dc12e3b671:9092
[Sarama]: 2022/11/18 16:59:42 Successfully initialized new client
[Sarama]: 2022/11/18 16:59:42 Failed to connect to broker c6dc12e3b671:9092: dial tcp: lookup c6dc12e3b671: Temporary failure in name resolution
[Sarama]: 2022/11/18 16:59:42 Error while sending ApiVersionsRequest to broker c6dc12e3b671:9092: dial tcp: lookup c6dc12e3b671: Temporary failure in name resolution
[Sarama]: 2022/11/18 16:59:42 client/metadata fetching metadata for [message.sent] from broker localhost:9092
[Sarama]: 2022/11/18 16:59:42 Failed to connect to broker c6dc12e3b671:9092: dial tcp: lookup c6dc12e3b671: Temporary failure in name resolution
[Sarama]: 2022/11/18 16:59:42 Error while sending ApiVersionsRequest to broker c6dc12e3b671:9092: dial tcp: lookup c6dc12e3b671: Temporary failure in name resolution
[Err]: 2022/11/18 16:59:42 dial tcp: lookup c6dc12e3b671: Temporary failure in name resolution
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x40 pc=0x6fe132]
goroutine 1 [running]:
main.main()
/home/sebastian/Dokumenty/GO/test/kafka/consumer/main.go:46 +0x472
exit status 2
