C# error when connecting to docker instance of Kafka

Question

I have a C# project using kafka-net (James Roland). I have a docker compose file which runs kafka on my laptop. I can produce and consume messages with the kafka instance from windows command line without issue.

When I try producing messages to the topic in kafka from the C# project I get this error

UnresolvedHostnameException: Could not resolve the following hostname: c60e8e54a7ca

System.AggregateException
  HResult=0x80131500
  Message=One or more errors occurred.
  Source=mscorlib
  StackTrace:
   at System.Threading.Tasks.Task.ThrowIfExceptional(Boolean includeTaskCanceledExceptions)
   at System.Threading.Tasks.Task.Wait(Int32 millisecondsTimeout, CancellationToken cancellationToken)
   at System.Threading.Tasks.Task.Wait()
   at CSharpKafka.Form1.<>c__DisplayClass2_0.<button1_Click>b__0() in D:\Play\Kafka\CSharpKafka\Form1.cs:line 50
   at System.Threading.ThreadHelper.ThreadStart_Context(Object state)
   at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)
   at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)
   at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state)
   at System.Threading.ThreadHelper.ThreadStart()

  This exception was originally thrown at this call stack:
    KafkaNet.DefaultKafkaConnectionFactory.GetFirstAddress(string, KafkaNet.IKafkaLog)
    KafkaNet.DefaultKafkaConnectionFactory.Resolve(System.Uri, KafkaNet.IKafkaLog)
    KafkaNet.BrokerRouter.UpdateInternalMetadataCache.AnonymousMethod__d(KafkaNet.Protocol.Broker)
    System.Linq.Enumerable.WhereSelectListIterator<TSource, TResult>.MoveNext()
    KafkaNet.BrokerRouter.UpdateInternalMetadataCache(KafkaNet.Protocol.MetadataResponse)
    KafkaNet.BrokerRouter.RefreshTopicMetadata(string[])
    KafkaNet.BrokerRouter.GetTopicMetadata(string[])
    KafkaNet.BrokerRouter.SelectBrokerRoute(string, byte[])
    KafkaNet.Producer.ProduceAndSendBatchAsync.AnonymousMethod__2a(KafkaNet.TopicMessage)
    System.Linq.Enumerable.WhereSelectEnumerableIterator<TSource, TResult>.MoveNext()
    ...
    [Call Stack Truncated]

Inner Exception 1:
AggregateException: One or more errors occurred.

Inner Exception 2:
UnresolvedHostnameException: Could not resolve the following hostname: c60e8e54a7ca

The docker compose is

version: '2'

networks:
  app-tier:
    driver: bridge

services:
  zookeeper:
    container_name: zookeeper-server
    image: 'bitnami/zookeeper:latest'
    environment:
        - ALLOW_ANONYMOUS_LOGIN=yes
    networks:
        - app-tier
  kafka:
    container_name: kafka-server
    image: 'bitnami/kafka:latest'
    environment:
        - ALLOW_PLAINTEXT_LISTENER=yes
        - KAFKA_CFG_ZOOKEEPER_CONNECT=zookeeper-server:2181
    networks:
        - app-tier
    ports:
      - "9092:9092"

The C# code is

            /*
             Adapted from https://www.c-sharpcorner.com/article/apache-kafka-net-application/
             */
            Uri uri =  new Uri("http://127.0.0.1:9092");

            string topic = "chat-messages";
            string payload = textBox1.Text;
            var sendMessage = new Thread(() => {
                
                KafkaNet.Protocol.Message 
                    msg = new KafkaNet.Protocol.Message(payload);

                var options = new KafkaOptions(uri);
                var router = new BrokerRouter(options);
                var client = new Producer(router);

                client.SendMessageAsync(topic, new List<KafkaNet.Protocol.Message> { msg }).Wait();
            });
            sendMessage.Start();

The docker ps output which relates to the error message is

CONTAINER ID   IMAGE                      COMMAND                  CREATED          STATUS          PORTS                                                                      NAMES
c60e8e54a7ca   bitnami/kafka:latest       "/opt/bitnami/script…"   16 minutes ago   Up 16 minutes   0.0.0.0:9092->9092/tcp, :::9092->9092/tcp                                  kafka-server
6e97e28bd530   bitnami/zookeeper:latest   "/opt/bitnami/script…"   16 minutes ago   Up 16 minutes   2181/tcp, 2888/tcp, 3888/tcp, 8080/tcp                                     zookeeper-server

The commands I have used from powershell command line to prove the kafka instance is running are

Consuming Messages

docker exec -it kafka-server /opt/bitnami/kafka/bin/kafka-console-consumer.sh --topic chat-messages --from-beginning --bootstrap-server localhost:9092

Producing Messages

docker exec -it kafka-server /opt/bitnami/kafka/bin/kafka-console-producer.sh --topic chat-messages --bootstrap-server localhost:9092

Looking for some help in solving. Thanks in advance

I guess your C# code is not running in a container itself, right? — David Brabant, Oct 21 '21 at 08:58
I haven't actually tested this, but I *think* need to set a new env-var for Kafka (`KAFKA_CFG_ADVERTISED_LISTENERS=CLIENT://kafka:9092,EXTERNAL://localhost:9092`). From memory, this allows Kafka to advertise itself as "localhost" (instead of it's hostname) when being talked to from outside docker (i.e. from the host machine). See the section [Accessing Kafka with internal and external clients](https://hub.docker.com/r/bitnami/kafka/) which discusses this in more detail - you might need to add a couple of other env-vars like `KAFKA_CFG_LISTENERS` as well. — RB., Oct 21 '21 at 09:01
@DavidBrabant Yes the C# code currently isn't running in the docker set. It will be once it works. I will check out your suggestion. Thank you. — yogibear, Oct 21 '21 at 09:07
If you do that, running your C# code in a container, replace 127.0.0.1 by "kafka" your service name. — David Brabant, Oct 21 '21 at 09:17
It looks like I need to explore the environment variable angle much more. The first one you suggested means I no longer get the exception. I see this now "No connection to:http://127.0.0.1:9092/. Attempting to connect...". When trying to access with service name I get more fundamental failures before the send statement executes, in terms of routes etc. So I will bear in mind the service name aspect, but also look more at the environment variables. I will share the solution once I find it. Thanks @DavidBrabant — yogibear, Oct 21 '21 at 09:32
@DavidBrabant many thanks for pointing me in the right direction. — yogibear, Oct 21 '21 at 09:49

score 2 · Accepted Answer · answered Oct 21 '21 at 09:49

Thanks to @David Brabant pointing me in the right direction with regards to environment variables and the docker image info page, I have this working now.

The C# code works. The only difference is that I added the following environment variables to the docker compose file.

        - KAFKA_CFG_ADVERTISED_LISTENERS=PLAINTEXT://127.0.0.1:9092
        - KAFKA_BROKER_ID=1
        - KAFKA_CFG_LISTENERS=PLAINTEXT://:9092
        - KAFKA_CFG_ZOOKEEPER_CONNECT=zookeeper:2181

I removed also some stuff I don't think I need from the original docker compose file.

My docker compose now looks like this and I didn't have to change anything else. The changes were detailed in the web page https://hub.docker.com/r/bitnami/kafka/ and scroll down to the section "Kafka development setup example".

version: '2'

services:
  zookeeper:
    container_name: zookeeper-server
    image: 'bitnami/zookeeper:latest'
    environment:
        - ALLOW_ANONYMOUS_LOGIN=yes
  kafka:
    container_name: kafka-server
    image: 'bitnami/kafka:latest'
    environment:
        - ALLOW_PLAINTEXT_LISTENER=yes
        - KAFKA_CFG_ZOOKEEPER_CONNECT=zookeeper-server:2181
        - KAFKA_CFG_ADVERTISED_LISTENERS=PLAINTEXT://127.0.0.1:9092
        - KAFKA_BROKER_ID=1
        - KAFKA_CFG_LISTENERS=PLAINTEXT://:9092
        - KAFKA_CFG_ZOOKEEPER_CONNECT=zookeeper:2181
    depends_on:
      - zookeeper
    ports:
      - "9092:9092"

I have to admit I didn't appreciate the internal setup would affect the connection in this way.

My connection URI in the C# code is

Uri uri =  new Uri("http://127.0.0.1:9092");

Kafka is not an HTTP service. Remove the prefix – OneCricketeer Oct 21 '21 at 14:11 — OneCricketeer, Oct 21 '21 at 14:11

C# error when connecting to docker instance of Kafka

1 Answers1