RDD foreach method provides no results

Question

I am trying to understand how foreach method works. In my jupyter notebook, I tried :

def f(x): print(x)
a = sc.parallelize([1, 2, 3, 4, 5])
b = a.foreach(f)
print(type(b))
<class 'NoneType'>

I can execute that without any problem, but I don't have any output except the print(type(b)) part. The foreach doesn't return anything, just a none type. I do not know what foreach is supposed to do, and how to use it. Can you explain me what it is used for ?

desertnaut · Accepted Answer · 2018-01-19T11:07:24.150

4

foreach is an action, and does not return anything; so, you cannot use it as you do, i.e. assigning it to another variable like b = a.foreach(f). From Learning Spark, p. 41-42:

Adapting the simple example from the docs, run in a PySpark terminal:

>>> def f(x): print(x)
>>> a = sc.parallelize([1, 2, 3, 4, 5])
>>> a.foreach(f)
5
4
3
1
2

(NOTE: not sure about Jupyter, but the above code will not produce any print results in a Databricks notebook.)

You may also find the answers in this thread helpful.

edited Jan 19 '18 at 11:07

answered Jan 19 '18 at 10:56

desertnaut

57,590
26
140
166

appart from `print`, do you have any other concret usecase ? – Steven Jan 19 '18 at 11:12
@Steven No. My feeling is that it is not widely used, apart from cases such the ones described in the book excerpt above. – desertnaut Jan 19 '18 at 11:13

amitabh Srivastava · Answer 2 · 2023-01-28T06:45:12.337

I just use the following method and it is working perfectly under Jupyter Notebook with PySpark:

for row in RDD.toLocalIterator():
    print(row)

Actually it will convert your RDD into a generator object and then by using this generator object you can easily iterate over each element. OR you can first create a generator object and then use it in your loop like below:

genobj = data.toLocalIterator()
for row in genobj:
    print(row)

RDD foreach method provides no results

2 Answers2