How to import a PySpark dataframe from one Jupyter Notebook to another without converting it to csv?

Question

Let's say that I have a dataframe called spark_df in a Notebook called Notebook1 and I want to transfer it to a Notebook called Notebook2. Obviously I can't do "from Notebook1.ipynb import spark_df" and I can't convert it to csv because 1) it's too big and 2) I need a more direct approach.

I need to import it to another Notebook because after finishing processing and I try to do something, the kernel dies. So how can I import the spark_df to Notebook2 without converting it to csv?

You most likely have to save it in some format. Or you oule upload it to drive and download from there, or could save it in zip format and read zip — Shisui Otsutsuki, Aug 30 '22 at 08:56
https://stackoverflow.com/questions/31621414/share-data-between-ipython-notebooks does that help you? — Oivalf, Aug 30 '22 at 08:56
Are you sharing it with another notebook running on the same server? If so, the kernel will probably die in any case. If not, look into feather or pickle — Josh Friedlander, Aug 30 '22 at 09:03

score 0 · Answer 1 · answered Aug 30 '22 at 09:17

0

Since your csv is too big to move in and out of disk, you can stream data from one spark job to another. See Structured Streaming Programming Guide

answered Aug 30 '22 at 09:17

s510

2,271
11
18

How to import a PySpark dataframe from one Jupyter Notebook to another without converting it to csv?

1 Answers1