Kafka SASL/PLAIN Setup with SSL

Question

I'm trying to configure SASL/PLAIN with SSL in our Kafka environment. The SSL part is done, but I'm running into the below error when starting the Zookeeper.
Has anyone configured SASL/PLAIN with SSL for Zookeeper and broker ?

Server-Broker Config

broker.id=19
port=19001

#host.name=localhost
host.name=jcia9081

#advertised.host.name=11.72.29.153

num.network.threads=3
num.io.threads=8

socket.send.buffer.bytes=102400
socket.receive.buffer.bytes=102400
socket.request.max.bytes=104857600

log.dirs=/appsdata/osp/kafka/datastore1/AIN_TEST_SSL/kafka,/appsdata/osp/kafka/datastore2/AIN_TEST_SSL/kafka,/appsdata/osp/kafka/datastore3/AIN_TEST_SSL/kafka

num.partitions=1
num.recovery.threads.per.data.dir=1

log.retention.hours=96
#log.retention.bytes=1073741824

log.segment.bytes=1073741824
log.retention.check.interval.ms=300000

log.cleaner.enable=false

replica.fetch.max.bytes=52428800

#The maximum size of a message that the server can receive.
message.max.bytes=52428800

zookeeper.connect=jcia9081:19101,jcia9131:19101,jcia9434:19101
zookeeper.connection.timeout.ms=6000
#################################
auto.create.topics.enable=false
auto.leader.rebalance.enable=true
delete.topic.enable=true
controlled.shutdown.enable=true

### SSL properties
security.inter.broker.protocol=SASL_SSL
listeners=SSL://jcia9081:19001,SASL_SSL://jcia9081:19002
ssl.keystore.location=/home/kafka/certs/jcia9081.federated.fds.jks
ssl.keystore.password=xxxx
ssl.key.password=xxxx
ssl.truststore.location=/home/kafka/certs/test.truststore.jks
ssl.truststore.password=xxx
sasl.mechanism.inter.broker.protocol=PLAIN
sasl.enabled.mechanisms=PLAIN
#listeners=SSL://jcia9081:19001
allow.everyone.if.no.acl.found=true

Zookeeper Config

dataDir=/appsdata/osp/kafka/datastore1/AIN_TEST_SSL/zookeeper

clientPort=19101

maxClientCnxns=0

tickTime=2000

initLimit=60
syncLimit=10

server.19=jcia9081:19121:19131
server.20=jcia9131:19121:19131
server.21=jcia9434:19121:19131

JAAS Zookeeper Configuration File

Server {
   org.apache.kafka.common.security.plain.PlainLoginModule required
   username="admin"
   password="admin-secret"
   user_admin="admin-secret"
};

JAAS Broker Configuration File

    KafkaServer {
        org.apache.kafka.common.security.plain.PlainLoginModule required
        username="admin"
        password="admin-secret"
        user_admin="admin-secret";
        };
Client {
        org.apache.kafka.common.security.plain.PlainLoginModule required
        username="admin"
        password="admin-secret";
};

Startup Error

[2017-04-26 10:05:24,775] INFO Client environment:user.dir=/appsdata/osp/kafka/AIN_TEST_SSL/startup (org.apache.zookeeper.ZooKeeper)
[2017-04-26 10:05:24,776] INFO Initiating client connection, connectString=jcia9081:19101 sessionTimeout=6000 watcher=org.I0Itec.zkclient.ZkClient@43195e57 (org.apache.zookeeper.ZooKeeper)
[2017-04-26 10:05:24,776] INFO Starting ZkClient event thread. (org.I0Itec.zkclient.ZkEventThread)
[2017-04-26 10:05:24,778] DEBUG zookeeper.disableAutoWatchReset is false (org.apache.zookeeper.ClientCnxn)
[2017-04-26 10:05:24,789] DEBUG JAAS loginContext is: Client (org.apache.zookeeper.client.ZooKeeperSaslClient)
[2017-04-26 10:05:24,795] DEBUG Awaiting connection to Zookeeper server (org.I0Itec.zkclient.ZkClient)
[2017-04-26 10:05:24,795] INFO Waiting for keeper state SaslAuthenticated (org.I0Itec.zkclient.ZkClient)
[2017-04-26 10:05:24,881] INFO successfully logged in. (org.apache.zookeeper.Login)
[2017-04-26 10:05:24,881] INFO Client will use DIGEST-MD5 as SASL mechanism. (org.apache.zookeeper.client.ZooKeeperSaslClient)
[2017-04-26 10:05:24,895] INFO Opening socket connection to server jcia9081.federated.fds/11.168.24.82:19101. Will attempt to SASL-authenticate using Login Context section 'Client' (org.apache.zookeeper.ClientCnxn)
[2017-04-26 10:05:24,968] INFO Socket connection established to jcia9081.federated.fds/11.168.24.82:19101, initiating session (org.apache.zookeeper.ClientCnxn)
[2017-04-26 10:05:24,970] DEBUG Session establishment request sent on jcia9081.federated.fds/11.168.24.82:19101 (org.apache.zookeeper.ClientCnxn)
[2017-04-26 10:05:25,001] INFO Session establishment complete on server jcia9081.federated.fds/11.168.24.82:19101, sessionid = 0x15baa8300580003, negotiated timeout = 6000 (org.apache.zookeeper.ClientCnxn)
[2017-04-26 10:05:25,002] DEBUG ClientCnxn:sendSaslPacket:length=0 (org.apache.zookeeper.client.ZooKeeperSaslClient)
[2017-04-26 10:05:25,004] DEBUG Received event: WatchedEvent state:SyncConnected type:None path:null (org.I0Itec.zkclient.ZkClient)
[2017-04-26 10:05:25,004] INFO zookeeper state changed (SyncConnected) (org.I0Itec.zkclient.ZkClient)
[2017-04-26 10:05:25,004] DEBUG Leaving process event (org.I0Itec.zkclient.ZkClient)
[2017-04-26 10:05:25,006] ERROR SASL authentication failed using login context 'Client'. (org.apache.zookeeper.client.ZooKeeperSaslClient)
[2017-04-26 10:05:25,006] DEBUG Received event: WatchedEvent state:AuthFailed type:None path:null (org.I0Itec.zkclient.ZkClient)
[2017-04-26 10:05:25,006] INFO zookeeper state changed (AuthFailed) (org.I0Itec.zkclient.ZkClient)
[2017-04-26 10:05:25,006] DEBUG Leaving process event (org.I0Itec.zkclient.ZkClient)
[2017-04-26 10:05:25,006] DEBUG Closing ZkClient... (org.I0Itec.zkclient.ZkClient)
[2017-04-26 10:05:25,006] INFO Terminate ZkClient event thread. (org.I0Itec.zkclient.ZkEventThread)
[2017-04-26 10:05:25,006] DEBUG Closing ZooKeeper connected to jcia9081:19101 (org.I0Itec.zkclient.ZkConnection)
[2017-04-26 10:05:25,006] DEBUG Close called on already closed client (org.apache.zookeeper.ZooKeeper)
[2017-04-26 10:05:25,006] DEBUG Closing ZkClient...done (org.I0Itec.zkclient.ZkClient)
[2017-04-26 10:05:25,007] FATAL Fatal error during KafkaServer startup. Prepare to shutdown (kafka.server.KafkaServer)
org.I0Itec.zkclient.exception.ZkAuthFailedException: Authentication failure
        at org.I0Itec.zkclient.ZkClient.waitForKeeperState(ZkClient.java:946)
        at org.I0Itec.zkclient.ZkClient.waitUntilConnected(ZkClient.java:923)
        at org.I0Itec.zkclient.ZkClient.connect(ZkClient.java:1230)
        at org.I0Itec.zkclient.ZkClient.<init>(ZkClient.java:156)
        at org.I0Itec.zkclient.ZkClient.<init>(ZkClient.java:130)
        at kafka.utils.ZkUtils$.createZkClientAndConnection(ZkUtils.scala:76)
        at kafka.utils.ZkUtils$.apply(ZkUtils.scala:58)
        at kafka.server.KafkaServer.initZk(KafkaServer.scala:327)
        at kafka.server.KafkaServer.startup(KafkaServer.scala:200)
        at kafka.server.KafkaServerStartable.startup(KafkaServerStartable.scala:39)
        at kafka.Kafka$.main(Kafka.scala:67)
        at kafka.Kafka.main(Kafka.scala)
[2017-04-26 10:05:25,009] INFO shutting down (kafka.server.KafkaServer)

Broker Start Command

    . /appsdata/osp/kafka/AIN_TEST_SSL/startup/setenv.sh
export JMX_PORT=19011
$KAFKA_HOME/bin/kafka-server-start.sh $KAFKA_CONFIG/server_AIN_TEST_SSL.properties > $LOG_DIR/kafka-broker-startup.log 2>&1 &

setenv

    export APP_HOME=/apps/osp

export JAVA_HOME=$APP_HOME/jdk1.8.0_101
export JAVA=$JAVA_HOME/bin/java

export KAFKA_HOME=$APP_HOME/kafka/v2.11-0.10.1.1
export KAFKA_CONFIG=/appsdata/osp/kafka/AIN_TEST_SSL/config

export ZK_HOME=$KAFKA_HOME
export ZK_CONFIG=$KAFKA_CONFIG

export LOG_DIR=/applogs/osp/kafka/AIN_TEST_SSL
export KAFKA_LOG4J_OPTS="-Dlog4j.configuration=file:$KAFKA_CONFIG/log4j.properties"

In Termninal 1

echo $KAFKA_OPTS
-Djava.security.auth.login.config=/appsdata/osp/kafka/AIN_TEST_SSL/config/jaas_AIN_TEST_SSL_zookeeper.conf

In Terminal 2

echo $KAFKA_OPTS
-Djava.security.auth.login.config=/appsdata/osp/kafka/AIN_TEST_SSL/config/jaas_AIN_TEST_SSL_server.conf

score 3 · Answer 1 · edited May 23 '17 at 10:31

3

You should give the path to your jaas file by exporting KAFKA_OPTS environment variable. Copy your jaas file under /appsdata/osp/kafka/AIN_TEST_SSL/config/

I assume, this is the content of your file:

kafka_server_jaas.conf

KafkaServer {
   org.apache.kafka.common.security.plain.PlainLoginModule required
   username="admin"
   password="admin-secret"
   user_admin="admin-secret";
};
Client {
   org.apache.kafka.common.security.plain.PlainLoginModule required
   username="admin"
   password="admin-secret";
};

zookeeper.properties

dataDir=/appsdata/osp/kafka/datastore1/AIN_TEST_SSL/zookeeper
clientPort=19101
maxClientCnxns=0
tickTime=2000
initLimit=60
syncLimit=10
server.19=jcia9081:19121:19131
server.20=jcia9131:19121:19131
server.21=jcia9434:19121:19131

#New configuration
authProvider.1=org.apache.zookeeper.server.auth.SASLAuthenticationProvider
requireClientAuthScheme=sasl
jaasLoginRenew=3600000

[BEGIN UPDATE]

Add the the zookeeper server jaas file and import it before starting zookeeper ensemble.

Server {
   org.apache.kafka.common.security.plain.PlainLoginModule required
   username="admin"
   password="admin-secret"
   user_admin="admin-secret"
};

After, export that jaas on the zookeeper terminal and start zookeeper

$ export KAFKA_OPTS="-Djava.security.auth.login.config=/appsdata/osp/kafka/AIN_TEST_SSL/config/zookeeper_jaas.conf"
$ bin/zookeeper-server-start.sh config/zookeeper.properties

Open a new terminal and hit the following commands to start your kafka server:

$ export KAFKA_OPTS="-Djava.security.auth.login.config=/appsdata/osp/kafka/AIN_TEST_SSL/config/kafka_server_jaas.conf"
$ bin/kafka-server-start.sh config/server.properties

[END UPDATE]

I think it will fix your issue.

Please look at my configurations on this post if any issue occurs:

Kafka SASL zookeeper authentication

edited May 23 '17 at 10:31

Community

1
1

answered Mar 20 '17 at 16:37

Maximilien Belinga

3,076
2
25
39

I still get the same error when I do "export KAFKA_OPTS" and start Zookeeper.I guess something else is off. – AtulyaB Mar 23 '17 at 16:10
Broker is not up. I'm trying to start the Zookeeper first. – AtulyaB Mar 23 '17 at 17:52
Ok sorry for the question I was far away and forgot kafka basics. So basically after exporting the jaas file you still have the same issue? You exported the zookeeper jaas in a terminal, and in the same one you tried to start kafka, right? – Maximilien Belinga Mar 23 '17 at 18:23
I first exported the JAAS file, and then in the same terminal, I started Zookeeper. Zookeeper never starts and dies with the error "ERROR No JAAS configuration section named 'Server' was foundin '/appsdata/osp/kafka/AIN_TEST_SSL/config/jaas_AIN_TEST_SSL_zookeeper.conf'" – AtulyaB Mar 23 '17 at 19:26
security.inter.broker.protocol=SASL_SSL listeners=SSL://jcia9082:19001,SASL_SSL://jcia9082:19002 ssl.keystore.location=/home/kafka/certs/jcia9081.federated.fds.jks ssl.keystore.password=xxx ssl.key.password=xxx ssl.truststore.location=/home/kafka/certs/test.truststore.jks ssl.truststore.password=welcome1 sasl.mechanism.inter.broker.protocol=PLAIN sasl.enabled.mechanisms=PLAIN – AtulyaB Mar 23 '17 at 19:28
Ok, please edit your post by adding your configuration files. both kafka server config file and zoo.cfg – Maximilien Belinga Mar 23 '17 at 19:37
Added broker and zookeeper configuration – AtulyaB Mar 23 '17 at 21:24
I updated my post. Basically, don't use a separate jaas file for zookeeper secured connection. And also, you should use `Client` instead of `Server` in jaas config file for Zookeeper. `Client` is used to authenticate zookeeper. – Maximilien Belinga Mar 24 '17 at 10:25
You are right when you say don't use separate JAAS file for zookeeper secured connection. I removed it. And I followed your instructions by configuring a JAAS file for broker only. I updated the post with the new startup error. Thank you for all the help so far. – AtulyaB Mar 24 '17 at 15:16
OK, the new error is when you try to start zookeeper? – Maximilien Belinga Mar 24 '17 at 15:26
Zookeeper starts without error. Its the Broker which gives the error. – AtulyaB Mar 24 '17 at 15:46
I think I know the issue. You have a syntax error on your jaas file. Look at what I just updated. I added ";" at the end of `password="admin-secret"` – Maximilien Belinga Mar 24 '17 at 15:46
Actually its a "Authentication failure "error. I updated the post. Is something missing in my broker config ? – AtulyaB Mar 24 '17 at 15:49
What is the full command do you use when starting the broker? – Maximilien Belinga Mar 24 '17 at 16:04
Also please bring back your configuration files. Maybe something is missing. Probably in `zookeeper.properties` file. Please update the post by adding the content of that config – Maximilien Belinga Mar 24 '17 at 16:10
As you enable authentication via SASL, you need to set the authentication provider, require sasl authentication and configure the login renewal period in `zookeeper.properties`. I just updated my post with the new `zookeeper.properties` file. update your config and restart zookeeper, then restart kafka – Maximilien Belinga Mar 24 '17 at 16:25
I think we are making progress. After I updated the zookeeper config, I started Zookeeper and then Broker. Zookeeper starts without error, but I'm still running into a "Authentication failure" during Broker startup. Updated post with error. – AtulyaB Mar 24 '17 at 18:36
Do you use a zookeeper cluster? – Maximilien Belinga Mar 24 '17 at 18:48
Yes. A 3 node cluster running on 3 VMs where one instance of zookeeper and broker are running on each VM . I've updated the Zookeeper config on all 3 nodes/VMs. But I'm starting Broker only on one VM. – AtulyaB Mar 24 '17 at 18:50
And do you export the jaas on the two other nodes? – Maximilien Belinga Mar 24 '17 at 18:53
yep.. I did. And when I try to start Broker on a different node, I get the same error. – AtulyaB Mar 24 '17 at 19:06
Please change the log level to `DEBUG` on your kafka log4j file and re-run the kafka server, then put the new stack trace. – Maximilien Belinga Mar 24 '17 at 19:13
Also, I think you should try to re-generate keytabs – Maximilien Belinga Mar 24 '17 at 19:44
I've enabled DEBUG mode in the log4j file and updated the error output. – AtulyaB Mar 27 '17 at 14:24
Ok I just updated the jaas file. I think the authentocation failed due to one line. Look at what I just added `user_admin="admin-secret"` in `KafkaServer` config. Try that and let me know please. Did you also re-generate certificates? – Maximilien Belinga Mar 27 '17 at 14:29
I updated the JAAS config file. While I did not get a authentication failure, I got a configuration error. I updated the post with the new error. No I did not regenerate the certificates. That is done by a different team and it usually takes time. Appreciate your help. – AtulyaB Mar 27 '17 at 15:16
Ok no worries about the certificates. To fix the config error you need to remove the ";" at this line `password="admin-secret";` and only that. But the ";" at the bottom need to stay of course. Look deeply at what I added – Maximilien Belinga Mar 27 '17 at 15:19
After making the update, I got a "Unable to open socket" error. I've update the post with the error and new JAAS server config – AtulyaB Mar 27 '17 at 15:32
Your zookeeper ensemble seems not working correctly. Please make sure zookeeper cluster nodes can communicate with each other and the ensemble is up and running – Maximilien Belinga Mar 27 '17 at 15:54
@M.Situation , any luck on solving the configuration issue, as i am facing the same Authentication Failure while trying to start kafka and my zookeeper is starting properly – sunder Apr 18 '17 at 10:14
You better have to ask to @AtulyaB, I don't know if it works for him. But everything should be already good. Also, do you use SSL as well? Maybe you should post your configuration by starting a new thread. Just notify me and I'll see how I can help you – Maximilien Belinga Apr 18 '17 at 10:39
@AtulyaB, Please let me know if the problem was solved for you. – sunder Apr 18 '17 at 10:51
@M.Situation, i have started a new thread :- http://stackoverflow.com/questions/43469962/kafka-sasl-zookeeper-authentication – sunder Apr 18 '17 at 10:51
@AtulyaB, please look at what I added on my post. It should be the solution to your issue. – Maximilien Belinga Apr 18 '17 at 14:23
@M.Situation Thank you for following up on the issue. Let me update my configuration with what you've added and I'll give you an update. – AtulyaB Apr 25 '17 at 13:36
I made changes to the Zookeeper JAAS file, and I'm getting the same zookeeper startup error I got before. I updated the post with the Zookeeper startup error and zookeeper JAAS file. – AtulyaB Apr 25 '17 at 17:43
Please try to setup environment variables before starting each technology. For instance, export zookeeper jaas before starting zookeeper in a terminal. In another terminal, export kafka jaas before starting kafka. I think it's looking for the Server entry in the bad file because of your env variables – Maximilien Belinga Apr 25 '17 at 18:38
I first exported the KAFKA_OPTS in terminal 1 for Zookeeper. This resolved the Zookeeper error. But when the export KAFKA_OPTS in terminal 2 for Kafka Broker, and start the process, I get an authentication failure. I don't know what I'm doing wrong here. I username matches to the both the zookeeper and broker JAAS file. I updated my post. Thank you for all the help. – AtulyaB Apr 26 '17 at 14:17
Also, I think I'm going to try the SASL setup without SSL and see if it works – AtulyaB Apr 26 '17 at 14:36
Ok do that first please, you can refer to @sunder post for that purpose: http://stackoverflow.com/questions/43469962/kafka-sasl-zookeeper-authentication – Maximilien Belinga Apr 26 '17 at 14:45
@M.Situation I did the SASL first and then integrated that SSL and it worked. Thank you for all the help. The next step for me is to configure SSL and SASL producers and consumers and establish connectivity and data flow. – AtulyaB Apr 28 '17 at 17:49
Ok you are welcome. If it answers your question please feel free to mark it as the answer! – Maximilien Belinga Apr 28 '17 at 19:10

Kafka SASL/PLAIN Setup with SSL

1 Answers1

Linked