1

i have python app that run with docker container that consume kafka, i also run kafka on my host machine, i run kafka manually not from docker.

when i run my python app container it won't connect to kafka from host, i run it with this command:

docker run -d --network="host" --name myptm-rating -p 8002:8002 myptm-rating-command

i set --network flag to connect my app with host.

this is my consumer configuration (i use confluent kafka package):

consumer_conf = {
        'bootstrap.servers': 'localhost:9092',
    }

then i see container logs, it shows errors:

[2020-11-23 04:09:44 +0000] [10] [INFO] Starting gunicorn 20.0.4
[2020-11-23 04:09:44 +0000] [10] [INFO] Listening at: http://0.0.0.0:8002 (10)
[2020-11-23 04:09:44 +0000] [10] [INFO] Using worker: uvicorn.workers.UvicornWorker
[2020-11-23 04:09:44 +0000] [13] [INFO] Booting worker with pid: 13
[2020-11-23 04:09:44 +0000] [14] [INFO] Booting worker with pid: 14
[2020-11-23 04:09:44 +0000] [15] [INFO] Booting worker with pid: 15
%3|1606104584.818|FAIL|rdkafka#producer-1| [thrd:localhost:9092/bootstrap]: localhost:9092/bootstrap: Connect to ipv6#[::1]:9092 failed: Connection refused (after 0ms in state CONNECT)
INFO:src.utils.kafka_client:Start kafka consumer...
%3|1606104584.826|FAIL|rdkafka#producer-1| [thrd:localhost:9092/bootstrap]: localhost:9092/bootstrap: Connect to ipv6#[::1]:9092 failed: Connection refused (after 0ms in state CONNECT)
%3|1606104584.827|FAIL|rdkafka#consumer-2| [thrd:localhost:9092/bootstrap]: localhost:9092/bootstrap: Connect to ipv4#127.0.0.1:9092 failed: Connection refused (after 0ms in state CONNECT)
[2020-11-23 04:09:44 +0000] [13] [INFO] Started server process [13]
[2020-11-23 04:09:44 +0000] [13] [INFO] Waiting for application startup.
[2020-11-23 04:09:44 +0000] [13] [INFO] Application startup complete.
INFO:src.utils.kafka_client:Start kafka consumer...
%3|1606104584.841|FAIL|rdkafka#consumer-2| [thrd:localhost:9092/bootstrap]: localhost:9092/bootstrap: Connect to ipv4#127.0.0.1:9092 failed: Connection refused (after 0ms in state CONNECT)
%3|1606104584.845|FAIL|rdkafka#producer-1| [thrd:localhost:9092/bootstrap]: localhost:9092/bootstrap: Connect to ipv4#127.0.0.1:9092 failed: Connection refused (after 0ms in state CONNECT)
INFO:src.utils.kafka_client:Start kafka consumer...
%3|1606104584.849|FAIL|rdkafka#consumer-2| [thrd:localhost:9092/bootstrap]: localhost:9092/bootstrap: Connect to ipv4#127.0.0.1:9092 failed: Connection refused (after 0ms in state CONNECT)
[2020-11-23 04:09:44 +0000] [14] [INFO] Started server process [14]
[2020-11-23 04:09:44 +0000] [14] [INFO] Waiting for application startup.
[2020-11-23 04:09:44 +0000] [14] [INFO] Application startup complete.
[2020-11-23 04:09:44 +0000] [15] [INFO] Started server process [15]
[2020-11-23 04:09:44 +0000] [15] [INFO] Waiting for application startup.
[2020-11-23 04:09:44 +0000] [15] [INFO] Application startup complete.
%3|1606104585.819|FAIL|rdkafka#producer-1| [thrd:localhost:9092/bootstrap]: localhost:9092/bootstrap: Connect to ipv6#[::1]:9092 failed: Connection refused (after 0ms in state CONNECT, 1 identical error(s) suppressed)
Mamen
  • 1,322
  • 5
  • 21
  • 44
  • you're opening port 8002 from the container but kafka is at port 9092 right? – OneRaynyDay Nov 23 '20 at 04:36
  • 8002 is my python app port @OneRaynyDay not kafka – Mamen Nov 23 '20 at 04:42
  • The docker config works the other way around. Your trying to connect to the host machine from the container right? Depending on your docker network configuration you might be able to point to host, as described here: https://stackoverflow.com/questions/24319662/from-inside-of-a-docker-container-how-do-i-connect-to-the-localhost-of-the-mach Alternatively you can make sure the Kafka on your host is reachable from a public fixed IP address or domain name. – JustLudo Nov 23 '20 at 07:15

1 Answers1

0

Make sure you connect on 0.0.0.0 instead of localhost, Docker can be very picky about these things.

consumer_conf = {
    'bootstrap.servers': '0.0.0.0:9092',
}
Gijs Wobben
  • 1,974
  • 1
  • 10
  • 13