0

I am running pyspark locally and had some issues due to something with the paths to python (when running python3 in command prompt I got an error, but when running python I would not. I have python 3 installed) I would get an java.io.IOException error when trying to run a pyspark job.

Now I have added

import os
import sys

os.environ['PYSPARK_PYTHON'] = sys.executable
os.environ['PYSPARK_DRIVER_PYTHON'] = sys.executable

which solves my problem. However, this does not seem like the best solution. Do I then in every file have to add this at the beginning or is there a smarter solution?

andKaae
  • 173
  • 1
  • 13
  • 1
    Does this answer your question? [PYCHARM Error-- java.io.IOException: Cannot run program "python3": CreateProcess error=2, The system cannot find the file specified](https://stackoverflow.com/questions/68705417/pycharm-error-java-io-ioexception-cannot-run-program-python3-createprocess) also see: [environment variables PYSPARK_PYTHON and PYSPARK_DRIVER_PYTHON](https://stackoverflow.com/questions/48260412/environment-variables-pyspark-python-and-pyspark-driver-python) you basically need to set those environment variables – Abdul Aziz Barkat Feb 27 '23 at 14:32
  • To be found the regular way, your Python 3.x must be in the `PATH` variable inherited in the environment of the process running pyspark. Not offering this as an answer because I don't know how you are set up so I can't describe a solution, but start reading with [this question](https://unix.stackexchange.com/questions/403972/the-python-command-starts-the-the-wrong-version-of-the-python-interpreter) and follow the trail from there. – alexis Feb 27 '23 at 14:35

2 Answers2

0

Just make a file with the code that fix the problem and import it in your main file.

since you import it in the main file it will execute the code imported first then run your code.

Curtis
  • 1
  • 1
0

Maybe try linking python3 to python?

PYTHON=$(which python)
sudo ln -s $PYTHON "$PYTHON"3

The ln command will generate a symbolic link to python in the same path where the shell is looking for python3.

Marc Sances
  • 2,402
  • 1
  • 19
  • 34