Is there a way to read from Parquet files in hdfs into SqlContext from Mobius?

Question

I know in Scala , you can read in a parquet file as follows:

//Create Spark Context
val sparkConf = new SparkConf().setAppName(appName).setMaster(sparkMaster)
val sc = new SparkContext(sparkConf)
val sqlContext = new SQLContext(sc)
import sqlContext.implicits._

 val pf = 
      sqlContext.read.parquet(hdfsDataUri + "test.parquet")
 pf.registerTempTable("test")

Is there a way to do this using Mobius (C# API for Spark)? I could only find a way to read in CSV files. Ref: https://github.com/Microsoft/Mobius

skaarthik · Accepted Answer · 2016-05-20T03:45:22.120

C# API for using Parquet in Apache Spark is available in Mobius. Following is the C# implementation of the Apache Spark Scala code in your question:

        var sparkConf = new SparkConf().SetAppName(appName).SetMaster(sparkMaster);
        var sc = new SparkContext(sparkConf);
        var sqlContext = new SqlContext(sc);
        var pf = sqlContext.Read().Parquet(hdfsDataUri + "test.parquet");
        pf.RegisterTempTable("test");

score 2 · Answer 2 · answered Nov 27 '17 at 23:22

2

You can read and write parquet files directly in .NET via https://github.com/elastacloud/parquet-dotnet

answered Nov 27 '17 at 23:22

Ivan G.

5,027
2
37
65

This is the better answer. Since Parquet.NET v3 release, arguably the best way to access Parquet. – Snympi Sep 16 '19 at 08:33

Is there a way to read from Parquet files in hdfs into SqlContext from Mobius?

2 Answers2