0

I use VS Code with the remote-containers extension.

I have 2 project folders for with each has it's own docker-compose file.

  1. zookeeper/kafka (listening to port 9092)
  2. python dev container (running as a remote-container)

In the python container I have a producer that sends messages to the kafka container. Unfortunately I cannot reach the kafka container from within the python container. I get the following error:
kafka.errors.NoBrokersAvailable: NoBrokersAvailable
(If I run the producer directly from the host then it works like a charm!)

How can I connect from the python remote-container to the kafka container?

I tried port forwarding and publishing in the devcontainer.json without success:
"forwardPorts": [9092]
"appPort": [9092] I even tried temporary port forwarding with Forward a port >> 9092

I'm a docker beginner so I guess I have a misconception here, I would highly appreciate if someone could help me out.

Thanks!
carlo

devcontainer.json

{
  "name": "python",
  "dockerComposeFile": ["../docker-compose.yml"],
  "service": "python",
  "workspaceFolder": "/workspace",
  "settings": {
    "terminal.integrated.shell.linux": "/bin/bash",
    "python.pythonPath": "/usr/local/bin/python",
    "python.linting.enabled": true,
    "python.linting.pylintEnabled": true,
    "python.linting.pylintPath": "/usr/local/bin/pylint",
    "python.formatting.autopep8Path": "/usr/local/bin/autopep8",
    "python.formatting.blackPath": "/usr/local/bin/black"
  },
  "extensions": [
    "ms-python.python",
    "coenraads.bracket-pair-colorizer",
    "wayou.vscode-todo-highlight"
  ],
  // forward ports
  "forwardPorts": [9092],
  // publish ports to the host
  "appPort": [9092]
}

producer.py

from kafka import KafkaProducer
import json
from data import get_registered_user
import time

def json_serializer(data):
    return json.dumps(data).encode('utf-8')

producer = KafkaProducer(
    bootstrap_servers=['localhost:9092'],
    value_serializer=json_serializer,
)

if __name__ == "__main__":
    while True:
        registered_user = get_registered_user()
        producer.send('registered_user', registered_user)
        print(registered_user)
        time.sleep(3)
CarloP
  • 99
  • 1
  • 12
  • Regardless of docker, if your producer is remote, why would you expect localhost:9092 to work? – OneCricketeer Oct 27 '20 at 13:59
  • I thought by forwarding port 9092 I was enabling the connection. As I said I think I have a misunderstanding here. Unfortunately this whole networking stuff always confuses me a bit... – CarloP Oct 28 '20 at 07:17
  • Unfortunately my question has been closed as a duplicate. I would like to point out that it's not a duplicate, because in the referred question the code runs from the host machine and in my case the code runs from a remote container as clearly described in the title and description. I would appreciate if the question would be reopened. – CarloP Oct 28 '20 at 07:26
  • That answer addresses what needs to be done to connect to a remote kafka container as well. Or you can read the blog linked at the top of the post – OneCricketeer Oct 28 '20 at 14:49
  • @OneCricketeer I'm sorry but I feel that I'm still misunderstood. I have no problem at all to connect to my remote kafka container (from my host machine). I have also read the blog post from Robin Moffatt, but I think it's not dealing with my particular issue either. My problem is that I'm not able to connect to the kafka container **from another remote docker container** in which I'm running VSCode. In other words, my problem is basically **how to connect to localhost from inside a docker container**. – CarloP Oct 30 '20 at 15:18
  • Luckily after more research I have found a solution by simply using `host.docker.internal` instead of `localhost` for the bootstrap_servers. I apologize if this information was also in your answer and I just couldn't understand it. In any case, thanks for you support! – CarloP Oct 30 '20 at 15:19
  • 1
    Answer is still the same. Advertised listeners need configured for the kafka container to be addressable on the network. If you using only one machine, advertising localhost on the same port that's forwarded, then it'll work. Otherwise, you need the DNS name / ip of the remote container – OneCricketeer Oct 30 '20 at 15:21
  • Well, that solution causes an extra network hop. The original problem is that `localhost` refers only to your python container. – OneCricketeer Oct 30 '20 at 15:23
  • 1
    Since you're using Docker Compose, you don't need `host.docker.internal` or forwarded ports from the host. You can use the default network bridge and access `kafka:9092` directly (as mentioned in the other post) – OneCricketeer Oct 30 '20 at 15:26
  • 1
    I think I finally got it. First I had **2 separate docker-compose files** in different folders. One for the docker/zookeeper containers and one for the python container. In that case I couldn’t reach the kafka container from the python container (unless I used host.docker.internal instead). Now I have setup a **dedicated bridge network** so that I can reach the kafka container. Another way would be to combine both the kafka/zookeeper containers and the python dev container in one docker-compose file so that I can directly access kafka without setting up a dedicated network. – CarloP Nov 01 '20 at 11:49
  • I guess my original question was just not precise enough, because my issue was not exactly to connect two containers, but rather to connect two containers that are not in the same network. @OneCricketeer Thanks for your support and patience! – CarloP Nov 01 '20 at 11:50
  • Yeah, sorry. Should've asked for the compose file. Glad you figured it out – OneCricketeer Nov 01 '20 at 14:24

0 Answers0