Kafka Listeners - how many?

Question

Update I'm updating this post to reflect the current configuration (and also, following @OneCricketeer response, more info)

According to this 2018 blog (which everyone seems to refer to) I am running Kafka (in a Docker Compose stack) with this configuration:

KAFKA_LISTENERS: DOCKER://kafka0:29092,LOCAL://localhost:9092
KAFKA_ADVERTISED_LISTENERS: DOCKER://kafka0:29092,LOCAL://localhost:9092
KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: DOCKER:PLAINTEXT,LOCAL:PLAINTEXT
KAFKA_INTER_BROKER_LISTENER_NAME: DOCKER

the Kafka broker should listen on both ports 29092 (used "within" the Docker network) as well as port 9092 (used by clients running on the host).

The problem is that it seems that the Kafka broker only responds on port 29092 when started like this:

    image: confluentinc/cp-kafka:7.0.1
    hostname: kafka
    container_name: kafka
    depends_on:
      - zookeeper
    ports:
      - "9092:9092"

(environment as above)

Using kafkacat this is what I get with the configuration above:

└─( nc -vz localhost 9092
Connection to localhost 9092 port [tcp/*] succeeded!

└─( kafkacat -b localhost:9092 -L 
% ERROR: Failed to acquire metadata: Local: Broker transport failure

Opening and connection on port 29092 works "as expected" only I would have to hack around /etc/hosts to make kafka0 point back to 127.0.0.1, which, as pointed out is A Really Bad Idea (hence the whole point of this question).

The diagrams and the text in that Confluent blog lead me to believe that either I'm missing something in the configuration of the container, or things have changed since 2018.

Answer 1

So I finally solved the riddle, thanks in part to this question.

The TL;DR version is that the host part of the listeners configuration has a different (opposite) meaning than it has for advertised.listeners: in the latter it means the host the client ought to talk to, in the former, the NIC address(es) the server should bind to.

In other words, changing the value of the host in listeners to be 0.0.0.0 (all available NICs) makes everything work:

KAFKA_LISTENERS: DOCKER://0.0.0.0:29092,LOCAL://0.0.0.0:9092
KAFKA_ADVERTISED_LISTENERS: DOCKER://kafka0:29092,LOCAL://localhost:9092
KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: DOCKER:PLAINTEXT,LOCAL:PLAINTEXT
KAFKA_INTER_BROKER_LISTENER_NAME: DOCKER

from the host machine (outside the container):

└─( kafkacat -b localhost:9092 -L 
Metadata for all topics (from broker 1: localhost:9092/1):
 1 brokers:
  broker 1 at localhost:9092 (controller)
 4 topics:
  topic "devices" with 1 partitions:
...

while from within the Docker network, it all works as before. For example:

{
  "@timestamp":"2021-12-25T00:05:27Z",
  "@service":"benthos",
  "component":"benthos.output",
  "level":"INFO",
  "message":"Sending Kafka messages to addresses: [kafka0:29092]"
}

and clients running on the host machine can connect to the Kafka cluster:

from pykafka import KafkaClient

client = KafkaClient(hosts='localhost:9092')
print(f"Available topics: {client.topics}")

# Produce or Consume data here

The full code sample is available here.

Worth noting that, AFAICT, the original blog post was giving an incorrect configuration for the listeners.

Answer 2

Never modify /etc/hosts to "fix" your listeners.

Nothing has changed until Kafka completely removes the need for Zookeeper.

lead me to believe that either I'm missing something in the configuration of the container

There are default listeners, but ports in Docker Compose has nothing to do with the actual broker server configuration. There are not two ports listening for external traffic. Only one for each internal and external.

Note that you are returned kafka0, as per your advertised setting, which is not the hostname you've set in your compose file, so your internal clients will have errors resolving hostname kafka0...

seems that Kafka only listens on port 29092 when started like this

It listens on both just fine.

$ nc -vz localhost 9092
Connection to localhost port 9092 [tcp/XmlIpcRegSvc] succeeded!
$ nc -vz localhost 29092
Connection to localhost port 29092 [tcp/*] succeeded!

The kafka protocol doesn't respond with the "bob" internal listener outside of the Docker network, no.

How many?

As many as necessary for your network. For example, I've seen up to four for internal replication using SASL, external clients using SSL, PLAINTEXT while we've migrated those clients to SSL, then later added SASL_SSL for more authorization controls.