1

Hi I'm trying to run the below example copied directly from Mobius Github. But when attempting to use the method SaveAsTextFile(string filepath) it's not finding the method. Instead its shows the error in Figure 1.

I'm referencing the latest released: "Microsoft.SparkCLR" version="1.6.100". 

 var lines = sparkContext.TextFile(@"hdfs://path/to/input.txt");  
    var words = lines.FlatMap(s => s.Split(' '));
    var wordCounts = words.Map(w => new KeyValuePair<string, int>(w.Trim(), 1))  
                          .ReduceByKey((x, y) => x + y);  
    var wordCountCollection = wordCounts.Collect();  
    wordCounts.SaveAsTextFile(@"hdfs://path/to/wordcount.txt");

Figure 1: does not contain definition....best alternative blah blah

zero323
  • 322,348
  • 103
  • 959
  • 935
raah
  • 158
  • 12

1 Answers1

2

Try the following line and it will work.

wordCounts.Map(wc => wc.Key + "," + wc.Value)
          .SaveAsTextFile(@"hdfs://path/to/wordcount.txt");

As the error message in your figure indicates SaveAsTextFile() is available if the RDD is of type string. The code above converts RDD of key-value pair into a string RDD. The code sample in the ReadMe file needs to be updated. Feel free to send a PR to fix it, if you are interested.

There was a discussion in Mobius project whether to make SaveAsTextFile() available for all RDD types. The problem with that is ToString on the type supported in the RDD may not always result in a readable string when written to the text file. If you have opinions on this feel free to create an issue in Mobius repo in GitHub.

skaarthik
  • 377
  • 2
  • 6