Method not implemented exception on Take method in Microsoft.Spark

Question

I am trying to setup spark with the new Microsoft.Spark library. The method DataFrame.PrintSchema works fine, however the method DataFrame.Take() gives an System.NotImplementedException. Allot of other methods also give this exception.

I took a look in the sources and that the 'Take' method calls the collect method and and it fails on the call to collectToPython.

SparkSession spark = SparkSession
    .Builder()
    .AppName(".NET Spark")
    .GetOrCreate();

DataFrame dataFrame = spark.Read().Json("people.json");
IEnumerable<Row> rows =  dataFrame.Take(1);

Is this just a Microsoft library that isn't finished yet? Or am I doing something wrong?

`Take` is an extension method for collections that will be available to you only when you add `System.Linq` as reference to your project(s). — Alberto Solano, May 10 '19 at 07:43

score 1 · Answer 1 · answered Jun 01 '19 at 03:16

Did you try the latest version released? I used v0.2.0 and the following works fine as expected:

var spark = SparkSession.Builder().GetOrCreate();
var df = spark.Read().Json("people.json");

IEnumerable<Row> rows = df.Take(1);
foreach (var row in rows)
{
    Console.WriteLine(row.Get("name"));
}
spark.Stop();

Method not implemented exception on Take method in Microsoft.Spark

1 Answers1