I was wondering if there is a way I can use C# to write queries to run on Apache spark. I know spark SQL queries can be written in java/scala/python. Is there any interface for c#?
Asked
Active
Viewed 3,749 times
6
-
what did searching for it online give ? – Shashank Shekhar Apr 28 '15 at 21:38
-
consider using https://github.com/spark-jobserver/spark-jobserver - and creating a rest interface in c# serialising json for data structures. I suggest this approach as jobserver provides context's which allow you to go back and re-use previous dataframes as well. – andrew.butkus Jan 04 '16 at 15:00
-
2https://github.com/dotnet/spark will do this. – cheerless bog Apr 26 '19 at 00:26
1 Answers
0
What exactly you are trying to achieve? If you want to write programs which uses spark apis, probably you are out of luck as only scala,java and python apis are exposed. But if you want to query spark storage data then you can see the thriftserver comes with spark and pass your queries through it. Essentially it should support any jdbc connection (see beeline example in documentation)
ayan guha
- 1,249
- 10
- 7
-
Would this work with Cassandra? Is there a way to pass an Spark SQL query through thriftserver to get back data without having to use Spark Jobs? – KingOfHypocrites Jun 16 '15 at 19:04