0

I'm using spark to read data files from hdfs.

When I do a spark action, a spark exception is raised:

Caused by: org.apache.hadoop.ipc.RemoteException(java.io.IOException): Delegation Token can be issued only with kerberos or web authentication

In the logs before the execption is thrown I can see:

 WARN   [hadoop.security.UserGroupInformation] PriviledgedActionException as:principal (auth:SIMPLE) cause:org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.ipc.StandbyException): Operation category READ is not supported in state standby. Visit https://s.apache.org/sbnn-error
[21/02/22 17:27:17.439] WARN   [hadoop.ipc.Client] Exception encountered while connecting to the server : org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.ipc.StandbyException): Operation category READ is not supported in state standby. Visit https://s.apache.org/sbnn-error
[21/02/22 17:27:17.440] WARN   [hadoop.security.UserGroupInformation] PriviledgedActionException as:principal (auth:SIMPLE) cause:org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.ipc.StandbyException): Operation category READ is not supported in state standby. Visit https://s.apache.org/sbnn-error

Which is really weird because I set the spark submit config with kerberos using --keytab and --principal configs:

spark.master                     yarn-cluster
spark.app.name                   app
spark.submit.deployMode          cluster
spark.yarn.principal             pincipal/principal.com
spark.yarn.keytab                all.keytab

spark.driver.memory              4G
spark.executor.memory            8G
spark.executor.instances         4
spark.executor.cores             8

spark.deploy.recoveryMode        ZOOKEEPER
spark.deploy.zookeeper.url       jdbc:phoenix:m1,m2,m3,m4:2181:/hbase
spark.driver.extraJavaOptions    -XX:MaxPermSize=1024M -Dlog4j.configuration=log4j.xml
spark.executor.extraJavaOptions  -Dlog4j.configuration=log4j.xml

I don't understand why the delegation token wouldn't be possible since the it is set up as kerberos auth.

I also don't understand why it displays those warnings as if the authentication mode of my spark was set as SIMPLE. Is spark ignoring my config ?

I have 2 environment, one on which the application works properly but I don't know what config I should look at.

Omegaspard
  • 1,828
  • 2
  • 24
  • 52
  • try to run 'kinit' command at the start of shell script – Arkon88 Feb 21 '22 at 16:58
  • Tying it right now, but wouldn't it be weird ? The spark submit is doing the kinit right ? – Omegaspard Feb 21 '22 at 17:03
  • we added kinit in ~/.bash_profile so once user logged it starts automatically. this is how it works in our case) – Arkon88 Feb 21 '22 at 17:08
  • Alright, doesn't work for me sadly. – Omegaspard Feb 21 '22 at 17:17
  • check https://stackoverflow.com/questions/44376334/how-to-fix-delegation-token-can-be-issued-only-with-kerberos-or-web-authenticat and try --conf spark.yarn.security.tokens.hive.enabled=false – Arkon88 Feb 21 '22 at 17:22
  • Sadly I checked this one, I'm not using hive – Omegaspard Feb 21 '22 at 17:44
  • `cluster` execution mode is a nightmare to debug. Start with `local` mode, then when it works bump to `yarn-client`, then when it works bump to `yarn-cluster` -- each level adds it share of issues. – Samson Scharfrichter Feb 21 '22 at 19:51
  • No, `spark-submit` does **not** run `kinit`. The principal/keytab params are consumed by the **driver**, to renew the initial kerberos creds in the case of long-running sessions _(e.g. streaming services)_. And in `cluster` mode the driver is not co-located with the client that bootstraps the whole job => in `local` or `client` modes the params might accidentally be used on start-up, but not in `cluster` mode – Samson Scharfrichter Feb 21 '22 at 19:56
  • For more background about Kerberos in Spark : https://stackoverflow.com/a/44290544/5162372 – Samson Scharfrichter Feb 21 '22 at 19:57
  • The spark-submit does manage the kerberos auth that is for sure. The issue I had is that the path I requested was hdfs://nodename:8020/pathToFile. Once I removed the node hdfs://nodedname:8020 it worked. – Omegaspard Feb 22 '22 at 07:22

0 Answers0