kerberos authentication in Kudu for spark2 job

Question

I am trying to put some data in kudu, but the worker cannot find the kerberos token, so I am not able to put some data into the kudu database.

here you can see my spark2-submit statement

spark2-submit --master yarn "spark.yarn.maxAppAttempts=1" --conf "spark.authenticate=true" --deploy-mode cluster --properties-file ./custom.properties --keytab /home/stream_ingest/stream_ingest.keytab  --principal xxx@xxx --class app.test.Task

the exception is the following:

18/06/08 08:33:02 INFO client.ConnectToCluster: Unable to connect to master xxx:7051: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available
18/06/08 08:33:02 INFO client.ConnectToCluster: Unable to connect to master xxx:7051: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available
18/06/08 08:33:02 INFO client.ConnectToCluster: Unable to connect to master xxx:7051: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available
ERROR: exception occured during an insert to kudu, sending record ConsumerRecord(topic = custom, partition = 0, offset = 120, CreateTime = 1528444250889, checksum = 3907376269, serialized key size = -1, serialized value size = 1067790, key = null, value = [B@1623bffa) to error topic custom
org.apache.kudu.client.NonRecoverableException: Couldn't find a valid master in (xxx:7051,xxx:7051,xxx:7051). Exceptions received: [org.apache.kudu.client.NonRecoverableException: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available, org.apache.kudu.client.NonRecoverableException: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available, org.apache.kudu.client.NonRecoverableException: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available]
    at org.apache.kudu.client.KuduException.transformException(KuduException.java:110)
    at org.apache.kudu.client.KuduClient.joinAndHandleException(KuduClient.java:352)
    at org.apache.kudu.client.KuduClient.openTable(KuduClient.java:222)
    at java.util.Iterator.forEachRemaining(Iterator.java:116)
    at org.apache.spark.api.java.JavaRDDLike$$anonfun$foreachPartition$1.apply(JavaRDDLike.scala:219)
    at org.apache.spark.api.java.JavaRDDLike$$anonfun$foreachPartition$1.apply(JavaRDDLike.scala:219)
    at org.apache.spark.rdd.RDD$$anonfun$foreachPartition$1$$anonfun$apply$29.apply(RDD.scala:926)
    at org.apache.spark.rdd.RDD$$anonfun$foreachPartition$1$$anonfun$apply$29.apply(RDD.scala:926)
    at org.apache.spark.SparkContext$$anonfun$runJob$5.apply(SparkContext.scala:2064)
    at org.apache.spark.SparkContext$$anonfun$runJob$5.apply(SparkContext.scala:2064)
    at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:87)
    at org.apache.spark.scheduler.Task.run(Task.scala:108)
    at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:338)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
    at java.lang.Thread.run(Thread.java:748)
    Suppressed: org.apache.kudu.client.KuduException$OriginalException: Original asynchronous stack trace
        at org.apache.kudu.client.ConnectToCluster.incrementCountAndCheckExhausted(ConnectToCluster.java:246)
        at org.apache.kudu.client.ConnectToCluster.access$100(ConnectToCluster.java:49)
        at org.apache.kudu.client.ConnectToCluster$ConnectToMasterErrCB.call(ConnectToCluster.java:365)
        at org.apache.kudu.client.ConnectToCluster$ConnectToMasterErrCB.call(ConnectToCluster.java:354)
        at com.stumbleupon.async.Deferred.doCall(Deferred.java:1280)
        at com.stumbleupon.async.Deferred.runCallbacks(Deferred.java:1259)
        at com.stumbleupon.async.Deferred.handleContinuation(Deferred.java:1315)
        at com.stumbleupon.async.Deferred.doCall(Deferred.java:1286)
        at com.stumbleupon.async.Deferred.runCallbacks(Deferred.java:1259)
        at com.stumbleupon.async.Deferred.callback(Deferred.java:1002)
        at org.apache.kudu.client.KuduRpc.handleCallback(KuduRpc.java:246)
        at org.apache.kudu.client.KuduRpc.errback(KuduRpc.java:300)
        at org.apache.kudu.client.RpcProxy.responseReceived(RpcProxy.java:221)
        at org.apache.kudu.client.RpcProxy.access$000(RpcProxy.java:60)
        at org.apache.kudu.client.RpcProxy$1.call(RpcProxy.java:132)
        at org.apache.kudu.client.RpcProxy$1.call(RpcProxy.java:128)
        at org.apache.kudu.client.Connection.cleanup(Connection.java:715)
        at org.apache.kudu.client.Connection.exceptionCaught(Connection.java:460)
        at org.apache.kudu.shaded.org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:112)
        at org.apache.kudu.client.Connection.handleUpstream(Connection.java:244)
        at org.apache.kudu.shaded.org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
        at org.apache.kudu.shaded.org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791)
        at org.apache.kudu.shaded.org.jboss.netty.channel.SimpleChannelUpstreamHandler.exceptionCaught(SimpleChannelUpstreamHandler.java:153)
        at org.apache.kudu.shaded.org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:112)
        at org.apache.kudu.shaded.org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
        at org.apache.kudu.shaded.org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791)
        at org.apache.kudu.shaded.org.jboss.netty.channel.SimpleChannelUpstreamHandler.exceptionCaught(SimpleChannelUpstreamHandler.java:153)
        at org.apache.kudu.shaded.org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:112)
        at org.apache.kudu.shaded.org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
        at org.apache.kudu.shaded.org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791)
        at org.apache.kudu.shaded.org.jboss.netty.handler.codec.oneone.OneToOneDecoder.handleUpstream(OneToOneDecoder.java:60)
        at org.apache.kudu.shaded.org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
        at org.apache.kudu.shaded.org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791)
        at org.apache.kudu.shaded.org.jboss.netty.handler.codec.frame.FrameDecoder.exceptionCaught(FrameDecoder.java:377)
        at org.apache.kudu.shaded.org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:112)
        at org.apache.kudu.shaded.org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
        at org.apache.kudu.shaded.org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559)
        at org.apache.kudu.shaded.org.jboss.netty.channel.Channels.fireExceptionCaught(Channels.java:525)
        at org.apache.kudu.shaded.org.jboss.netty.channel.AbstractChannelSink.exceptionCaught(AbstractChannelSink.java:48)
        at org.apache.kudu.shaded.org.jboss.netty.channel.DefaultChannelPipeline.notifyHandlerException(DefaultChannelPipeline.java:658)
        at org.apache.kudu.shaded.org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:566)
        at org.apache.kudu.shaded.org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791)
        at org.apache.kudu.shaded.org.jboss.netty.handler.timeout.ReadTimeoutHandler.messageReceived(ReadTimeoutHandler.java:184)
        at org.apache.kudu.shaded.org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70)
        at org.apache.kudu.shaded.org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
        at org.apache.kudu.shaded.org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791)
        at org.apache.kudu.shaded.org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:296)
        at org.apache.kudu.shaded.org.jboss.netty.handler.codec.oneone.OneToOneDecoder.handleUpstream(OneToOneDecoder.java:70)
        at org.apache.kudu.shaded.org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
        at org.apache.kudu.shaded.org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791)
        at org.apache.kudu.shaded.org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:296)
        at org.apache.kudu.shaded.org.jboss.netty.handler.codec.frame.FrameDecoder.unfoldAndFireMessageReceived(FrameDecoder.java:462)
        at org.apache.kudu.shaded.org.jboss.netty.handler.codec.frame.FrameDecoder.callDecode(FrameDecoder.java:443)
        at org.apache.kudu.shaded.org.jboss.netty.handler.codec.frame.FrameDecoder.messageReceived(FrameDecoder.java:303)
        at org.apache.kudu.shaded.org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70)
        at org.apache.kudu.shaded.org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
        at org.apache.kudu.shaded.org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559)
        at org.apache.kudu.shaded.org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:268)
        at org.apache.kudu.shaded.org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:255)
        at org.apache.kudu.shaded.org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:88)
        at org.apache.kudu.shaded.org.jboss.netty.channel.socket.nio.AbstractNioWorker.process(AbstractNioWorker.java:108)
        at org.apache.kudu.shaded.org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:337)
        at org.apache.kudu.shaded.org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89)
        at org.apache.kudu.shaded.org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
        at org.apache.kudu.shaded.org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
        at org.apache.kudu.shaded.org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
        ... 3 more

It looks like that the kudu client does not find the keytab file. In the kudu documentation they mentioned, that you only have to specify the keytab and principal parameter.

If you execute a openTable on the KuduClient in the driver, everything works fine.

The executors have no access to the keytab. It's the driver that manages the Kerberos tickets and obatains Hadoop "auth tokens" -- for HDFS and YARN, for Hive Metastore, for HBase... and probably Cloudera also developed an extra "token" for Kudu. Alas, whenever Spark fails to create a token (or it is disabled by configuration) it fails silently. — Samson Scharfrichter, Jun 08 '18 at 19:19
Cf. https://stackoverflow.com/questions/44265562/spark-on-yarn-secured-hbase for details - and debugging hints. — Samson Scharfrichter, Jun 08 '18 at 19:24
review user and password or recreate your keytab because adding --keytab /../my.keytab AND --principal myUSER@MYREALM is enought — AntuanSoft, Mar 27 '19 at 10:43

score 0 · Answer 1 · answered May 16 '23 at 06:11

While integrating kudu with Spark, to run the spark application in client mode you need to do authentication using kinit only. If you run the spark application in cluster mode you must need to specify the keytab and principal to the spark-submit to authenticate.

In your case, please check the principal and keytab is correct by running the sample SparkPi example.

For more steps you can go through the following article:

https://community.cloudera.com/t5/tkb/workflowpage/tkb-id/CommunityArticles/article-id/6689

score 0 · Answer 2 · answered Jun 07 '23 at 08:29

0

Not sure but it might be due to this bug: https://issues.apache.org/jira/browse/KUDU-2679

answered Jun 07 '23 at 08:29

hey_rey

103
8

kerberos authentication in Kudu for spark2 job

2 Answers2