Configuring listener with "localhost" causes a failure to retrieve meta data about the broker

Question

I am trying to set up a single Kafka broker using a docker container and I'm using the image from here. The objective is to run the Kafka broker in docker, and run the Producer/Consumers from the Host.

While configuring the KAFKA_LISTENERS and KAFKA_ADVERTISED_LISTENERS properties, I realised that I am unable to retrieve broker meta data if the following configuration is in place

KAFKA_BROKER_ID: 0
KAFKA_INTER_BROKER_LISTENER_NAME: INTERNAL
KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: INTERNAL:PLAINTEXT,EXTERNAL:PLAINTEXT
# KAFKA_ADVERTISED_LISTENERS must be EQUAL or SUBSET of KAFKA_LISTENERS
KAFKA_LISTENERS: INTERNAL://kafka0:29092, EXTERNAL://localhost:9092
# INTERNAL://kafka0:29092 is specified because of the KAFKA_INTER_BROKER_LISTENER_NAME configuration.
KAFKA_ADVERTISED_LISTENERS: INTERNAL://kafka0:29092, EXTERNAL://localhost:9092
KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181
KAFKA_AUTO_CREATE_TOPICS_ENABLE: 'false'
KAFKA_MESSAGE_MAX_BYTES: '200000'

> kafkacat -b localhost:9092 -L
ERROR: Failed to acquire metadata: Local: Broker transport failure

But replacing EXTERNAL://localhost:9092 with EXTERNAL://kafka0:9092 or EXTERNAL://:9092 (default interface) in the KAFKA_LISTENERS config fixes it.

>kafkacat -b localhost:9092 -L
Metadata for all topics (from broker 0: localhost:9092/0):
 1 brokers:
  broker 0 at localhost:9092 (controller)
 0 topics:

This is highly puzzling. Nothing in the Kafka documentation for Kafka 2.2.x says anything about this behaviour. http://kafka.apache.org/22/documentation.html#config. Hope someone can shed some light.

Full docker-compose file below

version: '2'
services:
  zookeeper:
    image: wurstmeister/zookeeper
    ports:
      - "2181:2181"
  kafka0:
    image: wurstmeister/kafka:2.12-2.2.1
    ports:
      - "9092:9092"
    environment:
      KAFKA_BROKER_ID: 0
      KAFKA_INTER_BROKER_LISTENER_NAME: INTERNAL
      KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: INTERNAL:PLAINTEXT,EXTERNAL:PLAINTEXT
      # KAFKA_ADVERTISED_LISTENERS must be EQUAL or SUBSET of KAFKA_LISTENERS
      KAFKA_LISTENERS: INTERNAL://kafka0:29092, EXTERNAL://localhost:9092
      # INTERNAL://kafka0:29092 is specified because of the KAFKA_INTER_BROKER_LISTENER_NAME configuration.
      KAFKA_ADVERTISED_LISTENERS: INTERNAL://kafka0:29092, EXTERNAL://localhost:9092
      KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181
      KAFKA_AUTO_CREATE_TOPICS_ENABLE: 'false'
      KAFKA_MESSAGE_MAX_BYTES: '200000'
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock

Verbose output for error

kafkacat -b localhost:9092 -L -d broker
%7|1564230563.245|BROKER|rdkafka#producer-1| [thrd:app]: localhost:9092/bootstrap: Added new broker with NodeId -1
%7|1564230563.245|BRKMAIN|rdkafka#producer-1| [thrd:localhost:9092/bootstrap]: localhost:9092/bootstrap: Enter main broker thread
%7|1564230563.245|CONNECT|rdkafka#producer-1| [thrd:app]: localhost:9092/bootstrap: Selected for cluster connection: bootstrap servers added (broker has 0 connection attempt(s))
%7|1564230563.245|BRKMAIN|rdkafka#producer-1| [thrd::0/internal]: :0/internal: Enter main broker thread
%7|1564230563.245|INIT|rdkafka#producer-1| [thrd:app]: librdkafka v1.1.0 (0x10100ff) rdkafka#producer-1 initialized (builtin.features gzip,snappy,ssl,sasl,regex,lz4,sasl_gssapi,sasl_plain,sasl_scram,plugins,zstd,sasl_oauthbearer, CC CXX PKGCONFIG OSXLD LIBDL PLUGINS ZLIB SSL SASL_CYRUS ZSTD HDRHISTOGRAM LZ4_EXT SNAPPY SOCKEM SASL_SCRAM SASL_OAUTHBEARER CRC32C_HW, debug 0x2)
%7|1564230563.245|CONNECT|rdkafka#producer-1| [thrd:app]: Not selecting any broker for cluster connection: still suppressed for 49ms: application metadata request
%7|1564230563.245|CONNECT|rdkafka#producer-1| [thrd:app]: Not selecting any broker for cluster connection: still suppressed for 49ms: application metadata request
%7|1564230563.245|CONNECT|rdkafka#producer-1| [thrd:localhost:9092/bootstrap]: localhost:9092/bootstrap: Received CONNECT op
%7|1564230563.245|STATE|rdkafka#producer-1| [thrd:localhost:9092/bootstrap]: localhost:9092/bootstrap: Broker changed state INIT -> TRY_CONNECT
%7|1564230563.245|CONNECT|rdkafka#producer-1| [thrd:localhost:9092/bootstrap]: localhost:9092/bootstrap: broker in state TRY_CONNECT connecting
%7|1564230563.245|STATE|rdkafka#producer-1| [thrd:localhost:9092/bootstrap]: localhost:9092/bootstrap: Broker changed state TRY_CONNECT -> CONNECT
%7|1564230563.245|CONNECT|rdkafka#producer-1| [thrd:app]: Not selecting any broker for cluster connection: still suppressed for 49ms: application metadata request
%7|1564230563.245|CONNECT|rdkafka#producer-1| [thrd:app]: Not selecting any broker for cluster connection: still suppressed for 49ms: application metadata request
%7|1564230563.246|CONNECT|rdkafka#producer-1| [thrd:localhost:9092/bootstrap]: localhost:9092/bootstrap: Connecting to ipv6#[::1]:9092 (plaintext) with socket 9
%7|1564230563.246|CONNECT|rdkafka#producer-1| [thrd:localhost:9092/bootstrap]: localhost:9092/bootstrap: Connected to ipv6#[::1]:9092
%7|1564230563.246|CONNECTED|rdkafka#producer-1| [thrd:localhost:9092/bootstrap]: localhost:9092/bootstrap: Connected (#1)
%7|1564230563.246|FEATURE|rdkafka#producer-1| [thrd:localhost:9092/bootstrap]: localhost:9092/bootstrap: Updated enabled protocol features +ApiVersion to ApiVersion
%7|1564230563.246|STATE|rdkafka#producer-1| [thrd:localhost:9092/bootstrap]: localhost:9092/bootstrap: Broker changed state CONNECT -> APIVERSION_QUERY
%7|1564230563.246|CONNECT|rdkafka#producer-1| [thrd:app]: Not selecting any broker for cluster connection: still suppressed for 48ms: application metadata request
%7|1564230563.246|CONNECT|rdkafka#producer-1| [thrd:app]: Not selecting any broker for cluster connection: still suppressed for 48ms: application metadata request
%7|1564230563.247|BROKERFAIL|rdkafka#producer-1| [thrd:localhost:9092/bootstrap]: localhost:9092/bootstrap: failed: err: Local: Broker transport failure: (errno: Connection reset by peer)
%7|1564230563.247|FEATURE|rdkafka#producer-1| [thrd:localhost:9092/bootstrap]: localhost:9092/bootstrap: Updated enabled protocol features -ApiVersion to
%7|1564230563.247|STATE|rdkafka#producer-1| [thrd:localhost:9092/bootstrap]: localhost:9092/bootstrap: Broker changed state APIVERSION_QUERY -> DOWN
%7|1564230563.247|BROKERFAIL|rdkafka#producer-1| [thrd:localhost:9092/bootstrap]: localhost:9092/bootstrap: failed: err: Local: Broker transport failure: (errno: Connection reset by peer)
%7|1564230563.247|FAIL|rdkafka#producer-1| [thrd:localhost:9092/bootstrap]: localhost:9092/bootstrap: ApiVersionRequest failed: Local: Broker transport failure: probably due to old broker version (after 0ms in state DOWN)
%7|1564230563.247|CONNECT|rdkafka#producer-1| [thrd:app]: Not selecting any broker for cluster connection: still suppressed for 47ms: application metadata request
%7|1564230563.247|CONNECT|rdkafka#producer-1| [thrd:app]: Not selecting any broker for cluster connection: still suppressed for 47ms: application metadata request
%7|1564230564.247|CONNECT|rdkafka#producer-1| [thrd:main]: Cluster connection already in progress: no cluster connection
%7|1564230564.248|STATE|rdkafka#producer-1| [thrd:localhost:9092/bootstrap]: localhost:9092/bootstrap: Broker changed state DOWN -> INIT
%7|1564230564.248|CONNECT|rdkafka#producer-1| [thrd:app]: Not selecting any broker for cluster connection: still suppressed for 48ms: application metadata request
%7|1564230564.248|CONNECT|rdkafka#producer-1| [thrd:app]: Not selecting any broker for cluster connection: still suppressed for 48ms: application metadata request
%7|1564230565.249|CONNECT|rdkafka#producer-1| [thrd:main]: localhost:9092/bootstrap: Selected for cluster connection: no cluster connection (broker has 1 connection attempt(s))
%7|1564230565.249|CONNECT|rdkafka#producer-1| [thrd:localhost:9092/bootstrap]: localhost:9092/bootstrap: Received CONNECT op
%7|1564230565.249|STATE|rdkafka#producer-1| [thrd:localhost:9092/bootstrap]: localhost:9092/bootstrap: Broker changed state INIT -> TRY_CONNECT
%7|1564230565.249|CONNECT|rdkafka#producer-1| [thrd:localhost:9092/bootstrap]: localhost:9092/bootstrap: broker in state TRY_CONNECT connecting
%7|1564230565.249|STATE|rdkafka#producer-1| [thrd:localhost:9092/bootstrap]: localhost:9092/bootstrap: Broker changed state TRY_CONNECT -> CONNECT
%7|1564230565.249|CONNECT|rdkafka#producer-1| [thrd:app]: Not selecting any broker for cluster connection: still suppressed for 49ms: application metadata request
%7|1564230565.249|CONNECT|rdkafka#producer-1| [thrd:app]: Not selecting any broker for cluster connection: still suppressed for 49ms: application metadata request
%7|1564230565.249|CONNECT|rdkafka#producer-1| [thrd:app]: Not selecting any broker for cluster connection: still suppressed for 49ms: application metadata request
%7|1564230565.249|CONNECT|rdkafka#producer-1| [thrd:app]: Not selecting any broker for cluster connection: still suppressed for 49ms: application metadata request
%7|1564230565.250|CONNECT|rdkafka#producer-1| [thrd:localhost:9092/bootstrap]: localhost:9092/bootstrap: Connecting to ipv6#[::1]:9092 (plaintext) with socket 9
%7|1564230565.250|CONNECT|rdkafka#producer-1| [thrd:localhost:9092/bootstrap]: localhost:9092/bootstrap: Connected to ipv6#[::1]:9092
%7|1564230565.250|CONNECTED|rdkafka#producer-1| [thrd:localhost:9092/bootstrap]: localhost:9092/bootstrap: Connected (#2)
%7|1564230565.250|FEATURE|rdkafka#producer-1| [thrd:localhost:9092/bootstrap]: localhost:9092/bootstrap: Updated enabled protocol features +ApiVersion to ApiVersion
%7|1564230565.250|STATE|rdkafka#producer-1| [thrd:localhost:9092/bootstrap]: localhost:9092/bootstrap: Broker changed state CONNECT -> APIVERSION_QUERY
%7|1564230565.250|CONNECT|rdkafka#producer-1| [thrd:app]: Not selecting any broker for cluster connection: still suppressed for 48ms: application metadata request
%7|1564230565.250|CONNECT|rdkafka#producer-1| [thrd:app]: Not selecting any broker for cluster connection: still suppressed for 48ms: application metadata request
%7|1564230565.254|BROKERFAIL|rdkafka#producer-1| [thrd:localhost:9092/bootstrap]: localhost:9092/bootstrap: failed: err: Local: Broker transport failure: (errno: Connection reset by peer)
%7|1564230565.254|FEATURE|rdkafka#producer-1| [thrd:localhost:9092/bootstrap]: localhost:9092/bootstrap: Updated enabled protocol features -ApiVersion to
%7|1564230565.254|STATE|rdkafka#producer-1| [thrd:localhost:9092/bootstrap]: localhost:9092/bootstrap: Broker changed state APIVERSION_QUERY -> DOWN
%7|1564230565.254|BROKERFAIL|rdkafka#producer-1| [thrd:localhost:9092/bootstrap]: localhost:9092/bootstrap: failed: err: Local: Broker transport failure: (errno: Connection reset by peer)
%7|1564230565.254|FAIL|rdkafka#producer-1| [thrd:localhost:9092/bootstrap]: localhost:9092/bootstrap: ApiVersionRequest failed: Local: Broker transport failure: probably due to old broker version (after 0ms in state DOWN)
%7|1564230565.254|CONNECT|rdkafka#producer-1| [thrd:app]: Not selecting any broker for cluster connection: still suppressed for 45ms: application metadata request
%7|1564230565.254|STATE|rdkafka#producer-1| [thrd:localhost:9092/bootstrap]: localhost:9092/bootstrap: Broker changed state DOWN -> INIT
%7|1564230565.254|CONNECT|rdkafka#producer-1| [thrd:app]: Not selecting any broker for cluster connection: still suppressed for 44ms: application metadata request
%7|1564230565.254|CONNECT|rdkafka#producer-1| [thrd:app]: Not selecting any broker for cluster connection: still suppressed for 44ms: application metadata request
%7|1564230565.254|CONNECT|rdkafka#producer-1| [thrd:app]: Not selecting any broker for cluster connection: still suppressed for 44ms: application metadata request
%7|1564230566.252|CONNECT|rdkafka#producer-1| [thrd:main]: localhost:9092/bootstrap: Selected for cluster connection: no cluster connection (broker has 2 connection attempt(s))
%7|1564230566.252|CONNECT|rdkafka#producer-1| [thrd:localhost:9092/bootstrap]: localhost:9092/bootstrap: Received CONNECT op
%7|1564230566.252|STATE|rdkafka#producer-1| [thrd:localhost:9092/bootstrap]: localhost:9092/bootstrap: Broker changed state INIT -> TRY_CONNECT
%7|1564230566.252|CONNECT|rdkafka#producer-1| [thrd:localhost:9092/bootstrap]: localhost:9092/bootstrap: broker in state TRY_CONNECT connecting
%7|1564230566.252|STATE|rdkafka#producer-1| [thrd:localhost:9092/bootstrap]: localhost:9092/bootstrap: Broker changed state TRY_CONNECT -> CONNECT
%7|1564230566.252|CONNECT|rdkafka#producer-1| [thrd:app]: Not selecting any broker for cluster connection: still suppressed for 49ms: application metadata request
%7|1564230566.252|CONNECT|rdkafka#producer-1| [thrd:app]: Not selecting any broker for cluster connection: still suppressed for 49ms: application metadata request
%7|1564230566.253|CONNECT|rdkafka#producer-1| [thrd:localhost:9092/bootstrap]: localhost:9092/bootstrap: Connecting to ipv6#[::1]:9092 (plaintext) with socket 9
%7|1564230566.253|CONNECT|rdkafka#producer-1| [thrd:localhost:9092/bootstrap]: localhost:9092/bootstrap: Connected to ipv6#[::1]:9092
%7|1564230566.253|CONNECTED|rdkafka#producer-1| [thrd:localhost:9092/bootstrap]: localhost:9092/bootstrap: Connected (#3)
%7|1564230566.253|FEATURE|rdkafka#producer-1| [thrd:localhost:9092/bootstrap]: localhost:9092/bootstrap: Updated enabled protocol features +ApiVersion to ApiVersion
%7|1564230566.253|STATE|rdkafka#producer-1| [thrd:localhost:9092/bootstrap]: localhost:9092/bootstrap: Broker changed state CONNECT -> APIVERSION_QUERY
%7|1564230566.253|CONNECT|rdkafka#producer-1| [thrd:app]: Not selecting any broker for cluster connection: still suppressed for 48ms: application metadata request
%7|1564230566.253|CONNECT|rdkafka#producer-1| [thrd:app]: Not selecting any broker for cluster connection: still suppressed for 48ms: application metadata request
%7|1564230566.254|BROKERFAIL|rdkafka#producer-1| [thrd:localhost:9092/bootstrap]: localhost:9092/bootstrap: failed: err: Local: Broker transport failure: (errno: Connection reset by peer)
%7|1564230566.254|FEATURE|rdkafka#producer-1| [thrd:localhost:9092/bootstrap]: localhost:9092/bootstrap: Updated enabled protocol features -ApiVersion to
%7|1564230566.254|STATE|rdkafka#producer-1| [thrd:localhost:9092/bootstrap]: localhost:9092/bootstrap: Broker changed state APIVERSION_QUERY -> DOWN
%7|1564230566.254|BROKERFAIL|rdkafka#producer-1| [thrd:localhost:9092/bootstrap]: localhost:9092/bootstrap: failed: err: Local: Broker transport failure: (errno: Connection reset by peer)
%7|1564230566.254|FAIL|rdkafka#producer-1| [thrd:localhost:9092/bootstrap]: localhost:9092/bootstrap: ApiVersionRequest failed: Local: Broker transport failure: probably due to old broker version (after 0ms in state DOWN)
%7|1564230566.254|CONNECT|rdkafka#producer-1| [thrd:app]: Not selecting any broker for cluster connection: still suppressed for 47ms: application metadata request
%7|1564230566.254|CONNECT|rdkafka#producer-1| [thrd:app]: Not selecting any broker for cluster connection: still suppressed for 47ms: application metadata request
%7|1564230567.254|CONNECT|rdkafka#producer-1| [thrd:main]: Cluster connection already in progress: no cluster connection
%7|1564230567.254|STATE|rdkafka#producer-1| [thrd:localhost:9092/bootstrap]: localhost:9092/bootstrap: Broker changed state DOWN -> INIT
%7|1564230567.254|CONNECT|rdkafka#producer-1| [thrd:app]: Not selecting any broker for cluster connection: still suppressed for 49ms: application metadata request
%7|1564230567.254|CONNECT|rdkafka#producer-1| [thrd:app]: Not selecting any broker for cluster connection: still suppressed for 49ms: application metadata request
%7|1564230568.248|CONNECT|rdkafka#producer-1| [thrd:app]: localhost:9092/bootstrap: Selected for cluster connection: application metadata request (broker has 3 connection attempt(s))
%7|1564230568.248|CONNECT|rdkafka#producer-1| [thrd:app]: Not selecting any broker for cluster connection: still suppressed for 49ms: application metadata request
%7|1564230568.248|CONNECT|rdkafka#producer-1| [thrd:localhost:9092/bootstrap]: localhost:9092/bootstrap: Received CONNECT op
%7|1564230568.248|STATE|rdkafka#producer-1| [thrd:localhost:9092/bootstrap]: localhost:9092/bootstrap: Broker changed state INIT -> TRY_CONNECT
%7|1564230568.248|CONNECT|rdkafka#producer-1| [thrd:localhost:9092/bootstrap]: localhost:9092/bootstrap: broker in state TRY_CONNECT connecting
%7|1564230568.248|STATE|rdkafka#producer-1| [thrd:localhost:9092/bootstrap]: localhost:9092/bootstrap: Broker changed state TRY_CONNECT -> CONNECT
% ERROR: Failed to acquire metadata: Local: Broker transport failure

Verbose output for expected output

kafkacat -b localhost:9092 -L -d broker
%7|1564230960.441|BROKER|rdkafka#producer-1| [thrd:app]: localhost:9092/bootstrap: Added new broker with NodeId -1
%7|1564230960.441|BRKMAIN|rdkafka#producer-1| [thrd::0/internal]: :0/internal: Enter main broker thread
%7|1564230960.441|BRKMAIN|rdkafka#producer-1| [thrd:localhost:9092/bootstrap]: localhost:9092/bootstrap: Enter main broker thread
%7|1564230960.441|CONNECT|rdkafka#producer-1| [thrd:app]: localhost:9092/bootstrap: Selected for cluster connection: bootstrap servers added (broker has 0 connection attempt(s))
%7|1564230960.441|INIT|rdkafka#producer-1| [thrd:app]: librdkafka v1.1.0 (0x10100ff) rdkafka#producer-1 initialized (builtin.features gzip,snappy,ssl,sasl,regex,lz4,sasl_gssapi,sasl_plain,sasl_scram,plugins,zstd,sasl_oauthbearer, CC CXX PKGCONFIG OSXLD LIBDL PLUGINS ZLIB SSL SASL_CYRUS ZSTD HDRHISTOGRAM LZ4_EXT SNAPPY SOCKEM SASL_SCRAM SASL_OAUTHBEARER CRC32C_HW, debug 0x2)
%7|1564230960.441|CONNECT|rdkafka#producer-1| [thrd:app]: Not selecting any broker for cluster connection: still suppressed for 49ms: application metadata request
%7|1564230960.441|CONNECT|rdkafka#producer-1| [thrd:app]: Not selecting any broker for cluster connection: still suppressed for 49ms: application metadata request
%7|1564230960.441|CONNECT|rdkafka#producer-1| [thrd:localhost:9092/bootstrap]: localhost:9092/bootstrap: Received CONNECT op
%7|1564230960.441|STATE|rdkafka#producer-1| [thrd:localhost:9092/bootstrap]: localhost:9092/bootstrap: Broker changed state INIT -> TRY_CONNECT
%7|1564230960.442|CONNECT|rdkafka#producer-1| [thrd:localhost:9092/bootstrap]: localhost:9092/bootstrap: broker in state TRY_CONNECT connecting
%7|1564230960.442|STATE|rdkafka#producer-1| [thrd:localhost:9092/bootstrap]: localhost:9092/bootstrap: Broker changed state TRY_CONNECT -> CONNECT
%7|1564230960.442|CONNECT|rdkafka#producer-1| [thrd:app]: Not selecting any broker for cluster connection: still suppressed for 49ms: application metadata request
%7|1564230960.442|CONNECT|rdkafka#producer-1| [thrd:app]: Not selecting any broker for cluster connection: still suppressed for 49ms: application metadata request
%7|1564230960.443|CONNECT|rdkafka#producer-1| [thrd:localhost:9092/bootstrap]: localhost:9092/bootstrap: Connecting to ipv6#[::1]:9092 (plaintext) with socket 9
%7|1564230960.443|CONNECT|rdkafka#producer-1| [thrd:localhost:9092/bootstrap]: localhost:9092/bootstrap: Connected to ipv6#[::1]:9092
%7|1564230960.443|CONNECTED|rdkafka#producer-1| [thrd:localhost:9092/bootstrap]: localhost:9092/bootstrap: Connected (#1)
%7|1564230960.443|FEATURE|rdkafka#producer-1| [thrd:localhost:9092/bootstrap]: localhost:9092/bootstrap: Updated enabled protocol features +ApiVersion to ApiVersion
%7|1564230960.443|STATE|rdkafka#producer-1| [thrd:localhost:9092/bootstrap]: localhost:9092/bootstrap: Broker changed state CONNECT -> APIVERSION_QUERY
%7|1564230960.443|CONNECT|rdkafka#producer-1| [thrd:app]: Not selecting any broker for cluster connection: still suppressed for 47ms: application metadata request
%7|1564230960.443|CONNECT|rdkafka#producer-1| [thrd:app]: Not selecting any broker for cluster connection: still suppressed for 47ms: application metadata request
%7|1564230960.448|FEATURE|rdkafka#producer-1| [thrd:localhost:9092/bootstrap]: localhost:9092/bootstrap: Updated enabled protocol features to MsgVer1,ApiVersion,BrokerBalancedConsumer,ThrottleTime,Sasl,SaslHandshake,BrokerGroupCoordinator,LZ4,OffsetTime,MsgVer2,IdempotentProducer,ZSTD
%7|1564230960.448|STATE|rdkafka#producer-1| [thrd:localhost:9092/bootstrap]: localhost:9092/bootstrap: Broker changed state APIVERSION_QUERY -> UP
%7|1564230960.451|CLUSTERID|rdkafka#producer-1| [thrd:main]: localhost:9092/bootstrap: ClusterId update "" -> "d3yPPxe8T6adHMiwXDT4Vg"
%7|1564230960.451|CONTROLLERID|rdkafka#producer-1| [thrd:main]: localhost:9092/bootstrap: ControllerId update -1 -> 0
%7|1564230960.451|UPDATE|rdkafka#producer-1| [thrd:localhost:9092/bootstrap]: localhost:9092/bootstrap: NodeId changed from -1 to 0
%7|1564230960.451|UPDATE|rdkafka#producer-1| [thrd:localhost:9092/bootstrap]: localhost:9092/0: Name changed from localhost:9092/bootstrap to localhost:9092/0
%7|1564230960.451|LEADER|rdkafka#producer-1| [thrd:localhost:9092/bootstrap]: localhost:9092/0: Mapped 0 partition(s) to broker
%7|1564230960.451|STATE|rdkafka#producer-1| [thrd:localhost:9092/bootstrap]: localhost:9092/0: Broker changed state UP -> UPDATE
%7|1564230960.451|STATE|rdkafka#producer-1| [thrd:localhost:9092/bootstrap]: localhost:9092/0: Broker changed state UPDATE -> UP
Metadata for all topics (from broker 0: localhost:9092/0):
 1 brokers:
  broker 0 at localhost:9092 (controller)
 0 topics:
%7|1564230960.453|DESTROY|rdkafka#producer-1| [thrd:app]: Terminating instance (destroy flags none (0x0))
%7|1564230960.453|DESTROY|rdkafka#producer-1| [thrd:main]: Destroy internal
%7|1564230960.453|DESTROY|rdkafka#producer-1| [thrd:main]: Removing all topics
%7|1564230960.453|DESTROY|rdkafka#producer-1| [thrd:main]: Sending TERMINATE to localhost:9092/0
%7|1564230960.453|TERM|rdkafka#producer-1| [thrd:localhost:9092/bootstrap]: localhost:9092/0: Received TERMINATE op in state UP: 1 refcnts, 0 toppar(s), 0 active toppar(s), 0 outbufs, 0 waitresps, 0 retrybufs
%7|1564230960.453|TERM|rdkafka#producer-1| [thrd::0/internal]: :0/internal: Received TERMINATE op in state INIT: 1 refcnts, 0 toppar(s), 0 active toppar(s), 0 outbufs, 0 waitresps, 0 retrybufs
%7|1564230960.453|BROKERFAIL|rdkafka#producer-1| [thrd::0/internal]: :0/internal: failed: err: Local: Broker handle destroyed: (errno: Undefined error: 0)
%7|1564230960.453|BROKERFAIL|rdkafka#producer-1| [thrd:localhost:9092/bootstrap]: localhost:9092/0: failed: err: Local: Broker handle destroyed: (errno: Resource temporarily unavailable)
%7|1564230960.453|FAIL|rdkafka#producer-1| [thrd::0/internal]: :0/internal: Client is terminating (after 11ms in state INIT)
%7|1564230960.453|STATE|rdkafka#producer-1| [thrd::0/internal]: :0/internal: Broker changed state INIT -> DOWN
%7|1564230960.453|TERMINATE|rdkafka#producer-1| [thrd::0/internal]: :0/internal: Handle is terminating in state DOWN: 1 refcnts (0x7fb393810b28), 0 toppar(s), 0 active toppar(s), 0 outbufs, 0 waitresps, 0 retrybufs: failed 0 request(s) in retry+outbuf
%7|1564230960.453|BROKERFAIL|rdkafka#producer-1| [thrd::0/internal]: :0/internal: failed: err: Local: Broker handle destroyed: (errno: Undefined error: 0)
%7|1564230960.453|FAIL|rdkafka#producer-1| [thrd:localhost:9092/bootstrap]: localhost:9092/0: Client is terminating (after 1ms in state UP)
%7|1564230960.453|STATE|rdkafka#producer-1| [thrd:localhost:9092/bootstrap]: localhost:9092/0: Broker changed state UP -> DOWN
%7|1564230960.453|TERMINATE|rdkafka#producer-1| [thrd:localhost:9092/bootstrap]: localhost:9092/0: Handle is terminating in state DOWN: 1 refcnts (0x7fb393811728), 0 toppar(s), 0 active toppar(s), 0 outbufs, 0 waitresps, 0 retrybufs: failed 0 request(s) in retry+outbuf
%7|1564230960.453|BROKERFAIL|rdkafka#producer-1| [thrd:localhost:9092/bootstrap]: localhost:9092/0: failed: err: Local: Broker handle destroyed: (errno: Resource temporarily unavailable)

OneCricketeer · Accepted Answer · 2020-04-21T04:59:59.197

8

For the listeners (not the advertised ones)

The default is 0.0.0.0, which means listening on all interfaces.

By setting it to a local hostname, it's only exposed locally (within the container), not to your host

edited Apr 21 '20 at 04:59

answered Jul 27 '19 at 15:18

OneCricketeer

179,855
19
132
245

But I don't understand how that affects the `KAFKA_ADVERTISED_LISTENERS` setting. Is it when I try to connect to localhost:9092, it will attempt to look for the listener bound at kafka0:9092? – cookiedookie Jul 27 '19 at 15:35
You don't an "external" regular listener because that's already how the setting works... you only need an advertised one. If you connect to port 9092, then it still responds with localhost:9092 – OneCricketeer Jul 27 '19 at 21:58
But that explains when I try to connect to localhost:9092 using `kafkacat -b localhost:9092 -L`, it is the `EXTERNAL://localhost:9092` listener on the `KAFKA_ADVERTISED_LISTENER` that i'm receiving meta data from. But how does having the same configuration on `KAFKA_LISTENER` prevents that from happening? – cookiedookie Jul 28 '19 at 02:36
As I explained, then it only listens on internal docker interfaces, not external ones, so the port forwards don't even work. You can see in the Confluent examples that the normal listeners aren't even set https://github.com/confluentinc/cp-docker-images/blob/5.3.0-post/examples/cp-all-in-one/docker-compose.yml – OneCricketeer Jul 28 '19 at 03:40
I am getting an error - `the nonroutable meta-address 0.0.0.0. Use a routable IP address.` when try to set 0.0.00 – Kanso Code Apr 21 '20 at 04:40
@Mikki `listeners` is different from `advertised.listeners`. Be sure to set the correct one – OneCricketeer Apr 21 '20 at 04:59

Configuring listener with "localhost" causes a failure to retrieve meta data about the broker

1 Answers1

Linked