1

I'm trying to implement Kafka producer in Go using Confluent-Kafka Client for Go, following the documentation here https://github.com/confluentinc/confluent-kafka-go#using-go-modules

My setup: Have a Lambda Function that reads Kafka Messages from Topic1 and sends out a different kafka Message to Topic2 after some processing.

For creating KafkaProducer, here's the code:

    producer, err := kafka.NewProducer(&kafka.ConfigMap{
        "bootstrap.servers": "host:port",
        "sasl.mechanisms":   "PLAIN",
        "security.protocol": "SASL_SSL",
        "sasl.jaas.config":  fmt.Sprintf("org.apache.kafka.common.security.plain.PlainLoginModule required username='%s' password='%s';", params.UserName, params.Password),
        "client.dns.lookup": "use_all_dns_ips"})

    if err != nil {
        fmt.Printf("Failed to create producer: %s", err)
        os.Exit(1)
    }

To send the kafka Message:

    err := producer.Produce(&kafka.Message{
        TopicPartition: kafka.TopicPartition{Topic: Topic2, Partition: kafka.PartitionAny},
        Key:            []byte(MyId),
        Value:          []byte(MyId),
    }, nil)
    if err != nil {
        fmt.Println("Error while sending message:", err)
        return
    }

To build this, I have a Makefile with command: env GOARCH=amd64 GOOS=linux go build -tags musl -ldflags="-s -w" -o bin/app cmd/app.go

make build throws error

internal/messageBus/createProducer.go:10:44: undefined: kafka.Producer
internal/messageBus/createProducer.go:14:25: undefined: kafka.NewProducer
internal/messageBus/createProducer.go:14:44: undefined: kafka.ConfigMap
make: *** [build] Error 2

I don't think I misread the documentation mentioned above, maybe something's missing in documentation??

Alex Pakka
  • 9,466
  • 3
  • 45
  • 69
flowAlong
  • 77
  • 10
  • Where did you get that build command? You should almost never be using filename arguments with the `go` tool. Just get the package building before trying to cross compile and set various flags. – JimB Nov 21 '22 at 17:52
  • @JimB I followed https://www.serverless.com/blog/framework-example-golang-lambda-support/ to try serverless with Go and that's where found that build command. As you suggested, I tried ``go build -tags musl cmd/app.go`` This shows no error. Adding ``env GOARCH=amd64 GOOS=linux`` throws errors mentioned in post – flowAlong Nov 21 '22 at 18:39
  • I suggest following the official documentation rather than random blog posts, filename arguments are only going to cause you trouble, just build the package. If the build fails only with GOARCH and GOOS set, then you have some build constraints conflicting with your intended build in some way. We need a more complete example to see what you are actually doing. – JimB Nov 21 '22 at 18:43
  • Thank you for suggestions @JimB. I see that Go toolchain does not support cross-compilation of CGo out of the box: https://github.com/confluentinc/confluent-kafka-go/issues/119 – flowAlong Nov 21 '22 at 18:44
  • cross-compiling cgo is supported "out of the box", but Go cannot include all possible compilers and libraries to cross-compile C code for your systems. – JimB Nov 21 '22 at 18:46
  • I'm trying to compile it from mac and now used Zig and works for me(https://dev.to/kristoff/zig-makes-go-cross-compilation-just-work-29ho) – flowAlong Nov 22 '22 at 11:58

0 Answers0