2

I have used Spark in Scala for a long time. Now I am using pyspark for the first time. This is on a Mac

  1. First I installed pyspark using conda install pyspark, and it installed pyspark 2.2.0
  2. I installed spark itself using brew install apache-spark, and it seems to have installed apache-spark 2.2.0

but when I run pyspark, it dumps out

/Users/me/anaconda/bin/pyspark: line 24: /Users/bruceho/spark-1.6/spark-1.6.2-bin-hadoop2.6/bin/load-spark-env.sh: No such file or directory
/Users/me/anaconda/bin/pyspark: line 77: /Users/bruceho/spark-1.6/spark-1.6.2-bin-hadoop2.6/bin/spark-submit: No such file or directory
/Users/me/anaconda/bin/pyspark: line 77: exec: /Users/bruceho/spark-1.6/spark-1.6.2-bin-hadoop2.6/bin/spark-submit: cannot execute: No such file

Why is it pointing to the 1.6.2 installation, which seems to be no longer there? Brew search apache-spark does indicate the presence of both 1.5. and 1.6. Shouldn't pyspark 2.2.0 automatically point to the apache-spark 2.2.0 installation?

Sukhi
  • 13,261
  • 7
  • 36
  • 53
bhomass
  • 3,414
  • 8
  • 45
  • 75

3 Answers3

4

There is a number of issues with your question:

To start with, PySpark is not an add-on package, but an essential component of Spark itself; in other words, when installing Spark you get also PySpark by default (you cannot avoid it, even if you would like to). So, step 2 should be enough (and even before that, PySpark should be available in your machine since you have been using Spark already).

Step 1 is unnecessary: Pyspark from PyPi (i.e. installed with pip or conda) does not contain the full Pyspark functionality; it is only intended for use with a Spark installation in an already existing cluster. From the docs:

The Python packaging for Spark is not intended to replace all of the other use cases. This Python packaged version of Spark is suitable for interacting with an existing cluster (be it Spark standalone, YARN, or Mesos) - but does not contain the tools required to setup your own standalone Spark cluster. You can download the full version of Spark from the Apache Spark downloads page.

NOTE: If you are using this with a Spark standalone cluster you must ensure that the version (including minor version) matches or you may experience odd errors

Based on the fact that, as you say, you have already been using Spark (via Scala), your issue seems rather to be about upgrading. Now, if you use pre-built Spark distributions, you have actually nothing to install - you just download, unzip, and set the relevant environment variables (SPARK_HOME etc) - see my answer on "upgrading" Spark, which is actually also applicable for first-time "installations".

desertnaut
  • 57,590
  • 26
  • 140
  • 166
  • I guess I did not make it clear. After removing old SPARK_HOME declarations in .bash_profile, everything is running. I have seen several documentations on installing Spark. Using homebrew is the by far the simplest, which is what I used. I already know the need for spark installation on top of pyspark. Like I said I installed the latest spark using brew. – bhomass Sep 25 '17 at 16:49
  • Indeed you did not, but anyway, hopefully you have learned something new (i.e. that step 1 is unnecessary) – desertnaut Sep 25 '17 at 17:24
  • I believe you can skip step 1 if you only use spark-submit. however, quite often, you would like to just run the spark code locally using python, in which case, you would want to have pyspark available to import. – bhomass Sep 30 '17 at 07:34
3

Step 1: If you don't have brew first install brew using the following command in terminal

/usr/bin/ruby -e "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/master/install)"

Step 2: Once you have brew then run below command to install java on your Mac.

brew cask install homebrew/cask-versions/adoptopenjdk8

Step 3: Once Java is installed run the below command to install spark on Mac

brew install apache-spark

Step 4: type pyspark -version

Dugini Vijay
  • 865
  • 7
  • 8
2

Easiest way to install pyspark right now would to do a pip install with version > 2.2.

pip install pyspark

If you want to use a distribution instead (and want to use jupyter along with it), another way would be: https://blog.sicara.com/get-started-pyspark-jupyter-guide-tutorial-ae2fe84f594f

Zoe
  • 27,060
  • 21
  • 118
  • 148
Kushagra Verma
  • 21
  • 1