2

I've written a very simple python script for testing my spark streaming idea, and plan to run it on my local machine to mess around a little bit. Here is the command line:

spark-submit spark_streaming.py localhost 9999

But the terminal threw me an error:

Error executing Jupyter command '<the/spark_streaming.py/file/path>': [Errno 2] No such file or directory

I have no idea why this would happen, and I'm sure the .py file does exist.

EDIT: there's no issue running it with python instead of spark-submit

And also, the lines added in the .bashrc file:

export PATH="/usr/local/spark/bin:$PATH"
export PYSPARK_DRIVER_PYTHON=jupyter
export PYSPARK_DRIVER_PYTHON_OPTS='notebook'
export SPARK_LOCAL_IP=localhost
James Wong
  • 1,107
  • 3
  • 15
  • 26

1 Answers1

3

Supposing you want to spark-submit to YARN a Python script located at /home/user/scripts/spark_streaming.py, the correct syntax is as follows:

spark-submit --master yarn --deploy-mode client /home/user/scripts/spark_streaming.py

You can interchange the ordering of the various flags, but the script itself must be at the end; if your script accepts arguments, they should follow the script name (e.g. see this example for calculating pi with 10 decimal digits).

For executing locally with, say, 2 cores, you should use --master local[2] - use --master local[*] for all available local cores (no deploy-mode flag in both cases).

Check the docs for more info (although admittedly they are rather poor in pyspark demonstrations).

PS The mention of Jupyter, as well the path shown in your error message are extremely puzzling...

UPDATE: Seems that PYSPARK_DRIVER_PYTHON=jupyter messes up everything, funneling the execution through Jupyter (which is undesirable here, and it may explain the weird error message). Try modifying the environment variables in your .bashrc as follows:

export SPARK_HOME="/usr/local/spark"  # do not include /bin
export PYSPARK_PYTHON=python
export PYSPARK_DRIVER_PYTHON=python
export PYSPARK_DRIVER_PYTHON_OPTS=""

and source .bashrc.

desertnaut
  • 57,590
  • 26
  • 140
  • 166
  • Unfortunately, no matter how I configure those flags and args, the error remain the same. Speaking of `Jupyter`, I guess it could be due to some environment variables I set in .bashrc file? And by the way I don't have any problem running it with `python` as opposed to `spark-submit`, which is weird. lol – James Wong Oct 16 '17 at 14:55
  • @JamesWong Jupyter should have nothing to do here. And where this `the/spark_streaming.py/file/path` comes from? Have you configured `SPARK_HOME`? – desertnaut Oct 16 '17 at 15:01
  • I didn't set SPARK_HOME anywhere explicitly as you can see in the updated post above. The spark_streaming.py file is simply the file sitting on the local disk. – James Wong Oct 16 '17 at 15:19
  • Thank you very much. That config solved the error. But on the flip side, I can't launch jupyter notebook by `pyspark` anymore. – James Wong Oct 17 '17 at 00:55
  • @JamesWong This way of using Jupyter with pyspark is obsolete & not recommended; kindly accept/upvote the answer if it resolved your issue, and possibly open another one with the new issue - my answer here could also be useful: https://stackoverflow.com/questions/46286021/how-to-use-jupyter-sparkr-and-custom-r-install/46346658#46346658 – desertnaut Oct 17 '17 at 12:12
  • I use jupyter notebook most of the time. So my cofing is set to jupyter and notebook. Making it to python solved the problem. But I have to revert it back to jupyter to use noteook again. – midtownguru Aug 12 '18 at 19:02
  • I have added SPARK_HOME to env in linus but still getting the same error. Any advice/help anyone? Thanks! – Drishti Jain Dec 08 '22 at 06:27