how to run sc.textFile() inside a foreach loop and do a union on it?

Question

so this is what I have been trying and I'm a newbie here working with spark!

I'm trying to execute this code

val ii=sc.parallelize(Seq(("e.txt"),("r.txt"))).foreach{i => sc.textFile(i)}

but I'm getting "Nullpointer exception"

Thanks!

score 1 · Accepted Answer · answered Dec 16 '16 at 06:49

1

You can just add multiple files to the sc.textFile. You should not use the sc inside of a map operation. The map function will be distributed to the different executors, and the sc lives in the driver. Therefore it will throw a Nullpointer exception.

a.txt contents:

a.txt:line1
a.txt:line2

b.txt contents:

b.txt:line1
b.txt:line2

Spark allows you to add more files in the same operation:

scala> sc.textFile("a.txt,b.txt").collect()
res1: Array[String] = Array(a.txt:line1, a.txt:line2, b.txt:line1, b.txt:line2)

Hope this helps and have fun with Spark!

answered Dec 16 '16 at 06:49

Fokko Driesprong

2,075
19
31

sc.textFile("a.txt,b.txt").collect() this syntax reads the file sequentially,, i need to do it in parallel fashion – Abu Tahir Dec 16 '16 at 07:35
It will execute in parallel. If you open the Spark UI you will see that two tasks are created. When you add more files, the number of tasks will increase. – Fokko Driesprong Dec 16 '16 at 07:46
i cant see any multiple task generated – Abu Tahir Dec 19 '16 at 09:46
What is your datasource? – Fokko Driesprong Dec 19 '16 at 14:30
can u take a look at this link? http://stackoverflow.com/questions/40286656/spark-executor-memory-exceeds-physical-limit?noredirect=1&lq=1 – Abu Tahir Dec 20 '16 at 10:09
I've posted my two cents :) – Fokko Driesprong Dec 21 '16 at 07:56
@AbuTahir zip files are not splittable and is a very bad input format. Perhaps you ment gzip? This is more common, but also not splittable. Please take a look at http://stackoverflow.com/questions/34208051/when-are-files-splittable – Fokko Driesprong Dec 21 '16 at 09:27
i'm trying to unzip 17 files and perform some processing , i want to unzip each of them as separate task just wanted to make thinks efficient. and suggest me a good book on spark to dig deep – Abu Tahir Dec 21 '16 at 16:20

how to run sc.textFile() inside a foreach loop and do a union on it?

1 Answers1