-2

I'm trying to use kafka on my local computer (Windows 10) as producer and a spark streaming on a docker container (ubuntu) as a consumer. My examples work together in the container but i need to create the stream on my local computer.

I'm running zookeeper at port 2181 in the host (local computer) and kafka server at 9092.

And i start a broker at 9092 to create the stream (local computer)

producer = KafkaProducer(bootstrap_servers='localhost:9092') while True: message='Hola' producer.send(1,'test', message) time.sleep(1)

On the other side in the container i'm listen to port 2181 to recieve the stream.

ssc = StreamingContext(sc, 5)

kvs = KafkaUtils.createStream(ssc, "127.0.0.1:2181", "spark-streaming-consumer", {'test': 1})

The problem is that i don't know how to link both (Windows Kafka and Ubuntu)

I hope you can help me. Thanks.

Victor Valero Fernandez
  • 55
  • 2
  • 8
  • see: http://stackoverflow.com/questions/17770902/forward-host-port-to-docker-container – jmunsch Mar 30 '17 at 23:36
  • 1
    This question lacks a lot of information. Can you post relevant commands on how you are starting the containers and where you need to "put on ip kafka parameters"? What is the host OS? Are you using Docker Machine? – Andy Shinn Mar 31 '17 at 00:35

1 Answers1

0

while starting the docker container expose port 8080 to 8080 and 7077 to 7077 , which can be done by providing -p 8080:8080 -p 7070:7070 in docker run command.

Then In kafka you can configure using localhost or 127.0.0.1

shahin
  • 3,515
  • 1
  • 17
  • 17