Link Spark with iPython Notebook

Question

I have followed some tutorial online but they do not work with Spark 1.5.1 on OS X El Capitan (10.11)

Basically I have run this commands download apache-spark

brew update
brew install scala
brew install apache-spark

updated the .bash_profile

# For a ipython notebook and pyspark integration
if which pyspark > /dev/null; then
  export SPARK_HOME="/usr/local/Cellar/apache-spark/1.5.1/libexec/"
  export PYSPARK_SUBMIT_ARGS="--master local[2]"
fi

run

ipython profile create pyspark

created a startup file ~/.ipython/profile_pyspark/startup/00-pyspark-setup.py configured in this way

# Configure the necessary Spark environment
import os
import sys

# Spark home
spark_home = os.environ.get("SPARK_HOME")

# If Spark V1.4.x is detected, then add ' pyspark-shell' to
# the end of the 'PYSPARK_SUBMIT_ARGS' environment variable
spark_release_file = spark_home + "/RELEASE"
if os.path.exists(spark_release_file) and "Spark 1.4" in open(spark_release_file).read():
    pyspark_submit_args = os.environ.get("PYSPARK_SUBMIT_ARGS", "")
    if not "pyspark-shell" in pyspark_submit_args: pyspark_submit_args += " pyspark-shell"
    os.environ["PYSPARK_SUBMIT_ARGS"] = pyspark_submit_args

# Add the spark python sub-directory to the path
sys.path.insert(0, spark_home + "/python")

# Add the py4j to the path.
# You may need to change the version number to match your install
sys.path.insert(0, os.path.join(spark_home, "python/lib/py4j-0.8.2.1-src.zip"))

# Initialize PySpark to predefine the SparkContext variable 'sc'
execfile(os.path.join(spark_home, "python/pyspark/shell.py"))

I then run ipython notebook --profile=pyspark and the notebook works fine, but the sc (spark context) is not recognised.

Anyone managed to do this with Spark 1.5.1?

EDIT: you can follow this guide to have it working

https://gist.github.com/tommycarpi/f5a67c66a8f2170e263c

Hi, I have the same problem and still unable to resolve. I use OS X Elcapitab, Jupiter, spark 1.5.2. What (extra) stuff did you do to make the "sc" work? — user1717931, Dec 17 '15 at 20:25
After solving all the issues I wrote this tutorial https://gist.github.com/tommycarpi/f5a67c66a8f2170e263c . Have a look at it, I used those exact steps and it's working fine. Hope this helps. — r4id4, Dec 17 '15 at 20:28

Alberto Bonsanto · Accepted Answer · 2017-04-07T17:29:33.507

I have Jupyter installed, and indeed It is simpler than you think:

Install anaconda for OSX.
Install jupyter typing the next line in your terminal Click me for more info.
```
ilovejobs@mymac:~$ conda install jupyter
```

Update jupyter just in case.

ilovejobs@mymac:~$ conda update jupyter

Download Apache Spark and compile it, or download and uncompress Apache Spark 1.5.1 + Hadoop 2.6.

ilovejobs@mymac:~$ cd Downloads 
ilovejobs@mymac:~/Downloads$ wget http://www.apache.org/dyn/closer.lua/spark/spark-1.5.1/spark-1.5.1-bin-hadoop2.6.tgz

Create an Apps folder on your home (i.e):

ilovejobs@mymac:~/Downloads$ mkdir ~/Apps

Move the uncompressed folder spark-1.5.1 to the ~/Apps directory.
```
ilovejobs@mymac:~/Downloads$ mv spark-1.5.1/ ~/Apps
```

Move to the ~/Apps directory and verify that spark is there.

ilovejobs@mymac:~/Downloads$ cd ~/Apps
ilovejobs@mymac:~/Apps$ ls -l
drwxr-xr-x ?? ilovejobs ilovejobs 4096 ?? ?? ??:?? spark-1.5.1

Here is the first tricky part. Add the spark binaries to your $PATH:

ilovejobs@mymac:~/Apps$ cd
ilovejobs@mymac:~$ echo "export $HOME/apps/spark/bin:$PATH" >> .profile

Here is the second tricky part. Add this environment variables also:

ilovejobs@mymac:~$ echo "export PYSPARK_DRIVER_PYTHON=ipython" >> .profile
ilovejobs@mymac:~$ echo "export PYSPARK_DRIVER_PYTHON_OPTS='notebook' pyspark" >> .profile

Source the profile to make these variables available for this terminal
```
ilovejobs@mymac:~$ source .profile
```
Create a ~/notebooks directory.
```
ilovejobs@mymac:~$ mkdir notebooks
```

Move to ~/notebooks and run pyspark:

ilovejobs@mymac:~$ cd notebooks
ilovejobs@mymac:~/notebooks$ pyspark

Notice that you can add those variables to the .bashrc located in your home. Now be happy, You should be able to run jupyter with a pyspark kernel (It will show it as a python 2 but it will use spark)

Thank you, I got it working without installing Jupiter (unless it was automagically installed :P ) — r4id4, Oct 11 '15 at 14:12
This worked for me...but now I can ONLY open the jupyter notebook. I no longer can just the python console using spark. Is it possble to have both? — Sebastialonso, Dec 03 '15 at 18:02
@Sebastialonso Yes it is possible, you just have to unset the `PYSPARK_DRIVER_PYTHON_OPTS` environment variable. — Alberto Bonsanto, Dec 03 '15 at 18:12
@AlbertoBonsanto followed the same steps. It gives an error Pyspark command not found. Any help? — Baktaawar, Aug 05 '16 at 23:17
echo "export PATH=/home/ilovejobs/Apps/spark/bin" >> .profile, this contains an important error since the PATH should be updated and not rewritten, otherwise will lead to serious issues. — marilena.oita, Apr 07 '17 at 14:44

score 4 · Answer 2 · answered Mar 20 '17 at 03:47

First, make sure you have got a spark enviornment in your machine.

Then, install a python module findspark via pip:

$ sudo pip install findspark

And then in the python shell:

import findspark
findspark.init()

import pyspark
sc = pyspark.SparkContext(appName="myAppName")

Now you can do what you want with pyspark in the python shell(or in ipython).

Actually it's the easiest way in my view to use spark kernel in the jupyter

Senkwich · Answer 3 · 2015-10-23T02:12:22.013

2

FYI, you can run Scala, PySpark, SparkR, and SQL with Spark running on top of Jupyter via https://github.com/ibm-et/spark-kernel now. The new interpreters were added (and marked experimental) from pull request https://github.com/ibm-et/spark-kernel/pull/146.

See the language support wiki page for more information.

edited Oct 23 '15 at 02:12

answered Oct 20 '15 at 03:57

Senkwich

1,022
1
9
17

score 1 · Answer 4 · answered Oct 12 '15 at 01:23

Spark with IPython/Jupyter notebook is great and I'm pleased the Alberto was able to help you get it working.

For reference it's also worth considering 2 good alternatives that come prepackaged and can easily be integrated into a YARN cluster (if desired.)

Spark Notebook: https://github.com/andypetrella/spark-notebook

Apache Zeppelin: https://zeppelin.incubator.apache.org/

At the time of writing Spark Notebook (v0.6.1) is more mature and you can prebuild an install against your Spark and Hadoop version here: http://spark-notebook.io/

Zeppelin (v0.5) looks very promising but doesn't offer as much functionality as Spark Notebook or IPython with Spark right now.

Link Spark with iPython Notebook

4 Answers4

Linked