Are there any python or scala tools to connect the spark/shark

Question

I want to use python or scala to connect shark server. But I didn't find any tools to do this. Are there any libs(python or scala/java). Thanks advanced.

score 1 · Answer 1 · edited May 23 '17 at 12:15

1

If you want to run SQL queries using Shark, Shark's sharkserver behaves like a regular Hive Thrift server, so you should be able to re-use existing Python methods for connecting to Hive, such as

Shark Server also supports Hive's JDBC interface, so you can use that to run queries from Scala or Java; just use the Shark Server's address in place of the Hive Server address.

edited May 23 '17 at 12:15

Community

1
1

answered Oct 12 '13 at 19:04

Josh Rosen

13,511
6
58
70

Thanks a lot!. That's what I want to try. – Joey.Chang Oct 24 '13 at 08:52

score 0 · Accepted Answer · answered Oct 12 '13 at 09:42

It is not clear what do you mean by connect but both shark and spark speak scala:

$./bin/shark-shell
scala> val youngUsers = sql2rdd("SELECT * FROM users WHERE age < 20")
scala> println(youngUsers.count)
...
scala> val featureMatrix = youngUsers.map(extractFeatures(_))
scala> kmeans(featureMatrix)

In addition spark speaks python as well.

Are there any python or scala tools to connect the spark/shark

2 Answers2