Connecting Kafka-Python with a cluster with Kerberos

Question

I'm trying to connect to a Kafka with Kafka-Python, the Kafka cluster has Kerberos that we need to build some commands to do few steps.

I have created one Topic at the cluster and I did the test with ./kafka-console-producer.sh and ./kafka-console-consumer.sh and works really well.

But when I try to connect with Kafka-Python I had a problem. See my code below:

def produce():
    print ('Producer')
    k_producer = KafkaProducer(bootstrap_servers='hostname:6667', 
                               security_protocol='SASL_PLAINTEXT',
                               sasl_plain_username='machine_usr',
                               sasl_plain_password='machine_pwd',
                               sasl_mechanism='PLAIN')
    for i in range(10):
        print ('Before send')
        k_producer.send('myTopic', 'Testing My Topic  ' + str(i))
        print ('After send')

Well, running this stuff I got this erro message after 30 secconds:

File "C:\Users\m\kafka-python-1.3.1\kafka.zip\kafka\producer\kafka.py", line 328, in __init__
File "C:\Users\m\kafka-python-1.3.1\kafka.zip\kafka\client_async.py", line 202, in __init__
File "C:\Users\m\kafka-python-1.3.1\kafka.zip\kafka\client_async.py", line 791, in check_version
kafka.errors.NoBrokersAvailable: NoBrokersAvailable

I'm running it in a remote machine. And the bootstrap_server I used the Zookeeper hostname and port but didn't work as well.

I found few things about it, and in the git of the Kafka-Python I found that they had implemented.

Is there other way to connect?

It would be helpful to see the contents of your working scripts. Also did you see this OP using a much simpler connection? http://stackoverflow.com/questions/35689238/kafka-python-producer-is-not-able-to-connect It might be a place to start, even if you have to set up a separate instance without Kerberos. — Daniel Wisehart, Sep 14 '16 at 14:23
Thanks @DanielWisehart! I know that we could be able to connect in a simple way. But the Kerberos make the things a little bit harder... I need to make an authentication with KeyTab... And it is not able to do that in python lib. — Thiago Baldim, Sep 26 '16 at 12:58
is the address of your kafka broker `hostname:6667` ? that doesn't seem right at first glance... — jimijazz, Oct 18 '16 at 17:12

score 5 · Answer 1 · answered Mar 13 '18 at 15:54

If the task is to solve this problem in python, another alternative could be to use confluent-kafka-python library that internally uses librdkafka that is written in C, and supports SASL, and the use of the keytab file. That wouldn't require having a separate Java process for the communication with kafka over SASL.

For instructions also refer to the documentation of the librdkafka library:

https://github.com/edenhill/librdkafka/wiki/Using-SASL-with-librdkafka - general intro https://github.com/edenhill/librdkafka/blob/master/CONFIGURATION.md - the properties that can be passed into the constructor of confluent_kafka.Producer and confluent_kafka.Consumer in python

Thiago Baldim · Accepted Answer · 2017-02-02T11:22:11.987

4

Well Guys,

I found the issue.

The problem is that Kerberos is not supported for Kafka producer in Python using Key Tab.

To use Key Tab we need to set a java Environment Variable.

According Hortonworks we need to set the client_jaas_client to connect.

The solution was using Py4j to call the Kafka Producer in JVM.

See the example here.

edited Feb 02 '17 at 11:22

answered Nov 08 '16 at 17:05

Thiago Baldim

7,362
3
29
51

Do you have an implementation example of this? Stuck in a similar situation. How do you use Py4j to call the Kafka Producer in JVM? – earl Jan 23 '21 at 18:14
Hey, this is the example I have used almost 4 years ago. That worked, unfortunately I don't have the original code :( https://gist.github.com/TRBaldim/2bf67f21bd31ceca3c44f12208a86be6 – Thiago Baldim Jan 26 '21 at 22:26

Connecting Kafka-Python with a cluster with Kerberos

2 Answers2