2

I am trying to run pyspark on windows. However, I received following error:

c:\spark>pyspark
Python 3.6.0 |Anaconda 4.3.1 (64-bit)| (default, Dec 23 2016, 11:57:41) [MSC 
v.1900 64 bit (AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.
Traceback (most recent call last):
File "C:\spark\bin\..\python\pyspark\shell.py", line 38, in <module>
   SparkContext._ensure_initialized()
File "C:\spark\python\pyspark\context.py", line 259, in _ensure_initialized
   SparkContext._gateway = gateway or launch_gateway(conf)
File "C:\spark\python\pyspark\java_gateway.py", line 80, in launch_gateway
   proc = Popen(command, stdin=PIPE, env=env)
File "C:\Users\shuzhe\Anaconda3\lib\subprocess.py", line 707, in __init__
   restore_signals, start_new_session)
File "C:\Users\shuzhe\Anaconda3\lib\subprocess.py", line 990, in _execute_child
startupinfo)
PermissionError: [WinError 5] Access is denied

I have searched for hours, but I could not find a solution to this problem. Any help is greatly appreciated!

Thanks!

Parker
  • 193
  • 2
  • 3
  • 14

2 Answers2

1

Hi I ran into the same problem. I ditched the SPARK_HOME folder (which in my case was C:\spark-2.4.3-bin-hadoop2.7) that I created previously using cygwin tar and tried unpacking the folder again (from spark-2.4.3-bin-hadoop2.7.tgz), but this time using 7zip instead of cygwin tar and then problem went away. Note that with 7zip the unpacking is done in two step, first extract the ...tar from ...tgz, and then extract the whole directory tree from ...tar

Some other discussion on the Internet (e.g. here) suggested that the problem was due to cygwin tar got some permission wrong. This explanation is consistent with what I tested and experienced above.

leeyuiwah
  • 6,562
  • 8
  • 41
  • 71
0

If you are using spark 2.1 you will have to switch to spark 2.1.1.There is a bug with python 3.6 and now is resolved in 2.1.1 .

https://issues.apache.org/jira/browse/SPARK-19019\

I had same problem with python 3.6 and 3.5 and spark 2.1 on windows , and I switched to anaconda python 2.7 and now it works fine.

zlidime
  • 1,124
  • 11
  • 6
  • Hi. Actually I used spark 2.1.1, and this problem still exists. So do you have any other ideas about what should i do? – Parker May 13 '17 at 22:45
  • Sorry , I installed 2.1.1 spark , with same anaconda version 4.3.1 ,windows 10 , and java 1.8.0_131-b11 and hadoop 2.7.3 ,winutils 2.7.1 and i could't get same error as you. – zlidime May 14 '17 at 10:18