End of file exception while reading a file from remote hdfs cluster using spark

Question

I am new to working with HDFS. I am trying to read a csv file which is stored in a hadoop cluster using spark. Every time i try to access it i get the following error: End of File Exception between local host

I have not setup hadoop locally since i already had access to hadoop cluster.

I may be missing some configurations but i dont know which ones. Would appreciate the help.

I tried to debug it using this : link

Did not work for me.

This is the code using spark.

val conf= new SparkConf().setAppName("Read").setMaster("local").set("fs.hdfs.impl", classOf[org.apache.hadoop.hdfs.DistributedFileSystem].getName)
      .set("fs.file.impl", classOf[org.apache.hadoop.fs.LocalFileSystem].getName)
val sc=new SparkContext(conf)
val data=sc.textfile("hdfs://<some-ip>/abc.csv)

I expect it to read the csv and convert it into RDD.

Getting this error: Exception in thread "main" java.io.EOFException: End of File Exception between local host is:

did you try this. https://stackoverflow.com/questions/27478096/cannot-read-a-file-from-hdfs-using-spark — Andy_101, Aug 27 '19 at 11:45
ya i have tried that. I am confused with where to set xml files if any needs to be added to configuration — Vaibhav Srivastava, Aug 27 '19 at 11:48
@VaibhavSrivastava - if this cluster is personal you can try once format name node — Chetan Hirapara, Aug 27 '19 at 11:55

score 0 · Answer 1 · answered Aug 27 '19 at 12:31

0

Run you spark jobs on hadoop cluster. Use below code:

val spark = SparkSession.builder().master("local[1]").appName("Read").getOrCreate()
val data = spark.sparkContext.textFile("<filePath>")

or you can use spark-shell as well.

If you want to access hdfs from your local, follow this:link

answered Aug 27 '19 at 12:31

wypul

807
6
9

I see the error comes when i try to perform action on the rdd returned. i am getting the error when i do data.count(). What could be the problem – Vaibhav Srivastava Aug 27 '19 at 12:43
@VaibhavSrivastava - this rdd's are lazy evolution so it is not perfomaining untill some action is triggred – Chetan Hirapara Aug 27 '19 at 12:45
ya so count() is an action right. When i try to perform count it is giving end of file error. How to resolve this – Vaibhav Srivastava Aug 27 '19 at 12:48

End of file exception while reading a file from remote hdfs cluster using spark

1 Answers1