16

I am getting the below error while running a pyspark program on PYCHARM, Error:

java.io.IOException: Cannot run program "python3": CreateProcess error=2, The system cannot find the file specified ......

The interpreter is recognizing the python.exe file and I have added the Content root in project structure.

I got a similar issue while running the same program before in on windows command prompt and solved it using What is the right way to edit spark-env.sh before running spark-shell?

James Z
  • 12,209
  • 10
  • 24
  • 44
KKS
  • 173
  • 1
  • 1
  • 5
  • Welcome to Stack Overflow. There are a few posts for approximately this error message see [pycharm cannot run program is:q](https://stackoverflow.com/search?tab=votes&q=pycharm%20cannot%20run%20program%20is%3aq). This one may be what you want [PyCharm error: Cannot run program, error=2, No such file or directory](https://stackoverflow.com/questions/63223548). However I think there isn't a thread about your exact error message. I'm assuming this is PySpark specific so any details you could add to the question would be helpful. – bad_coder Aug 09 '21 at 01:26

3 Answers3

28

Before creating your spark session, set the following environment variables in your code:

import os
import sys
from pyspark.sql import SparkSession

os.environ['PYSPARK_PYTHON'] = sys.executable
os.environ['PYSPARK_DRIVER_PYTHON'] = sys.executable
spark = SparkSession.builder.getOrCreate()
elyptikus
  • 936
  • 8
  • 24
23

create an environment variable PYSPARK_PYTHON with value 'python' or the path to your respective python executable.

Abdul Aziz Barkat
  • 19,475
  • 3
  • 20
  • 33
Nuka Tejeswara Rao
  • 241
  • 1
  • 5
8
  1. Go to Environmental variable and within System variable set a new variable as PYSPARK_PYTHON and value as python

PYSPARK_PYTHON=python

  1. Add below codebits to your pyspark code
import os
import sys
from pyspark import SparkContext
os.environ['PYSPARK_PYTHON'] = sys.executable
os.environ['PYSPARK_DRIVER_PYTHON'] = sys.executable
Vineeth Reddy
  • 198
  • 1
  • 8
  • This is the way to go. Also, after setting PYSPARK_PYTHON env var, you might need to restart whatever IDE / CMD Prompt you are using. – Vae Jiang Sep 14 '22 at 01:36