0

We have a single node kafka installation on AWS. We are able to run kafka producer and consumer in this environment. But while our client tried to send messages (outside AWS environment) to our kafka broker, their messages stuck at their firewall.

So client kafka producer is setup behind a firewall/proxy, how can our client send messages through kafka after bypassing the proxy at their end.

learner
  • 1
  • 1
  • 1

1 Answers1

2

First check that the problem is indeed a client firewall issue and not related to either incorrect broker settings (like using private AWS IP as advertised listener) or AWS security rules (not allowing external machines to connect on port 9092/9093). Easiest way to check is by publishing yourself from an external host without a firewall.

Next check that they have allowed TCP connections on port 9092 (or whatever you are using for Kafka) to pass through their client side firewall.

If it's not possible to open this port in the firewall then you might consider either sending the data to a TCP proxy (like haproxy) on port 80 or 443 since those posts are often open.

If that still doesn't work then consider publishing using an HTTP(S) proxy like the Confluent REST Proxy as it uses standard HTTP(S) ports and protocols that should traverse most client firewalls.

Hans Jespersen
  • 8,024
  • 1
  • 24
  • 31
  • We were able to publish data from an external host without a firewall successfully. Can you explain more how we can send the data to a TCP proxy (like haproxy) on port 80 or 443 – learner Jul 25 '17 at 15:02
  • 1
    Similar to this but listen (bind) to port 80 or 443 and then connect to 9092 on the backend https://stackoverflow.com/questions/45216724/how-to-load-balance-kafka-boostrap-with-haproxy but you will then have to make the advertised listener port in kafka server.properties configuration file to be the ip and port of the haproxy – Hans Jespersen Jul 26 '17 at 01:15
  • Can we set any proxy setting while publishing the messages to kafka broker. – learner Jul 26 '17 at 08:00
  • Not sure what you are asking. The Producer would have to point to the Haproxy server as if it was a Kafka server – Hans Jespersen Jul 26 '17 at 12:05
  • So we should create a HAproxy server between kafka producer and kafka Broker, right? – learner Jul 28 '17 at 07:03
  • Yes. You can also just try and run the Kafka broker SSL listen port on 443 and see if that works first. If the client firewall won't allow SSL on 443 then HaProxy won't work either. Just make sure the AWS security rules allow TCP connections on 443 from whatever the clients public IP address is. – Hans Jespersen Jul 28 '17 at 07:06
  • At client side 443 port is blocked. Is there any other way, they can connect to kafka broker . – learner Jul 28 '17 at 07:26
  • What isn't blocked? Can they at least do http on port 80? Then at least you could use the Confluent Kafka REST Proxy but the client code would have to use the REST API instead of Kafka Producer/Consumer – Hans Jespersen Jul 28 '17 at 07:31
  • Consider the Kafka REST Proxy then configured to listen on port 80. http://docs.confluent.io/current/kafka-rest/docs/index.html – Hans Jespersen Jul 28 '17 at 07:39
  • In this case, we need to write the code to publish data using Kafka REST proxy on client side and also to receive it at broker side, right? – learner Jul 28 '17 at 08:42
  • You can publish with REST API and subscribe with normal Java consumer API – Hans Jespersen Jul 28 '17 at 18:27