-4

How to convert org.apache.spark.rdd.RDD[(String, String)] = org.apache.spark.rdd.RDD[String, String] in Spark core with Scala 

INPUT
(2020-01-19,ERROR)
(2020-01-19,INFO)

OUTPUT
2020-01-19,ERROR
2020-01-19,INFO
DennisLi
  • 3,915
  • 6
  • 30
  • 66
Sholk
  • 3
  • 2
  • 3
    There is not such thing as a `RDD[String, String]`. What are you trying to accomplish? – Jasper-M Feb 01 '20 at 08:35
  • What about using dataframes? Did you try that out? – abiratsis Feb 01 '20 at 12:12
  • Also check [this](https://stackoverflow.com/questions/48427185/how-to-make-good-reproducible-apache-spark-examples) on how to write well defined reproducible Spark questions. – abiratsis Feb 01 '20 at 12:59
  • Step 1: Submit pull request to have RDDs take two type parameters (doing whatever it is you want it to do, which you did not say). Step 3: Profit. – Jack Leow Feb 01 '20 at 23:37

2 Answers2

1

How to convert org.apache.spark.rdd.RDD[(String, String)] = org.apache.spark.rdd.RDD[String, String] in Spark core with Scala

You can't convert to org.apache.spark.rdd.RDD[String, String], because as you can clearly see from the documentation of org.apache.spark.rdd.RDD[T] , it only takes one type parameter, not two. You can't convert to something that doesn't exist.

Jörg W Mittag
  • 363,080
  • 75
  • 446
  • 653
0

It looks like you come from Java spark. In Scala, the equivalent of JavaPairRDD<String, String> is simply RDD[(String, String)]: In Spark Scala, RDD[(K, V)]s are extended with pair RDD behaviors thanks to implicit conversion (see PairRDDFunctions)

bonnal-enzo
  • 1,165
  • 9
  • 19