0

I'm trying to read a file from hdfs using

var lines = Source.fromFile("hdfs://nameservice1/location/report_2018-06-03.csv").getLines.toArray

however I keep getting the error

java.io.FileNotFoundException: hdfs:/nameservice1/location/report_2018-06-03.csv (No such file or directory)

Is there a different command to read csv's from hdfs or is there a workaround for this?

Edit: Just a clarification, I just can't get it to look in hdfs://nameservice...

Foxlooo
  • 69
  • 2
  • 9
  • 3
    can be possible duplicate of https://stackoverflow.com/questions/41587931/read-the-data-from-hdfs-using-scala – sumitya Jul 09 '18 at 16:15
  • I cannot use that because when I get the input that way, it becomes an object instead of a string which I cannot split. – Foxlooo Jul 09 '18 at 17:54
  • 1
    `Source.fromFile` cannot read from HDFS. You must use Hadoop FileSystem object API. To get the name service working, you must place core-site.xml and hdfs-site.xml files from the cluster on the classpath. If you really want to process a CSV file, why aren't you using Spark? – OneCricketeer Jul 10 '18 at 00:48

0 Answers0