-1

How to read a .text file in edge node using Scala.

def main(args: Array[String]) {
  val srcFile=sc.textFile("file://home//viji.palanisamy//dev//kpi_library//EDI//Prof_test1").toString()
  readFile(srcFile)
}

def readFile(filename: String) = {
  val bufferedSource = Source.fromFile(filename)
  println("bufferedSource"+bufferedSource)
}

Got an error like file not found. Is there any solution to read text from node using Scala.

Community
  • 1
  • 1
vijay
  • 1
  • 9

1 Answers1

0

Is there any solution to read text from node using Scala.

Unless you want to read file locally, using standard IO tools (Read entire file in Scala?) and parallelize there isn't.

You have to distribute the file to each executor node or place it in a distributed storage.

The trick shown here https://stackoverflow.com/a/47845360/9658417 by user8371915:

import org.apache.spark.SparkFiles

val filename: String = ???
sc.addFile(filename)

val srcFile=sc.textFile(SparkFiles.get(filename))

However if all you want is to read local file on the driver, skip textFile part and only:

Source.fromFile(filename)