2

I am trying to connect to Hive(hadoop cluster has kerberos authentication) from Spark which is Standalone.

Can someone let me know how to do kinit in spark program i could connect to hive?

UPDATE:My Spark is on different cluster from Hadoop

AKC
  • 953
  • 4
  • 17
  • 46

2 Answers2

1

Assuming you have a spark-shell open and you don't want to exit, and then re-kinit you could do something like this:

import java.lang.ProcessBuilder
import java.io.PrintWriter

//resets your kerberos login
val p1 = Runtime.getRuntime.exec("kdestroy")
p1.waitFor
//executes kinit, 
val p = Runtime.getRuntime.exec("kinit")
val stdin = p.getOutputStream
val pw =new PrintWriter(stdin)
//val pwd = get_password() //get_password() is a function to get your password from a file, or wherever
pw.println(pwd) // you could put your password here , but plain text passwords are generally frowned upon
pw.close
p.waitFor
James Tobin
  • 3,070
  • 19
  • 35
  • 2
    Since Hive uses the Hadoop auth library, you can also use a single line of Scala: `org.apache.hadoop.security.UserGroupInformation.loginUserFromKeytab("the.donald@TRUMP.TOWER", "/some/path/to/the.donald.keytab")` – Samson Scharfrichter May 02 '17 at 13:44
  • Is it same if the Spark is on Diff cluster? – AKC May 02 '17 at 15:58
  • The whole point of Kerberos is to authenticate a *remote* client -- so you need a Kerberos conf that points to the KDC used by Hadoop, a Hadoop conf that points to the cluster, and Hadoop client (JARs + native libs). Plus canonical DNS names for the cluster nodes. With all that, you can run Spark from anywhere. Including from Windows -- but that's more tricky because of the lack of "official" native libs. – Samson Scharfrichter May 02 '17 at 21:55
0

you should run the kinit command before init the spark-shell or submit a spark app (spark-submit). The other option is get the kerberos ticket using a keytab and run your spark program like this

bash -c "kinit -kt /path/to/key/mykey.keytab myuser@KERBEROS.SERVER.COM; spark-shell"

regards,

hlagos
  • 7,690
  • 3
  • 23
  • 41