Error with pyspark in local when I execute pytest in VS Code from Git Bash

Question

I have the Error with pyspark in local when I execute pytest in VS Code from Git Bash.

If I execute pytest from gitbash console:

enter image description here

I try to debug my code when I create the spark Dataframe in Visual code, show me:

Java gateway process exited before sending its port number

enter image description here

I have configured all environment of variables in my PC:

enter image description here

I have seen this error can be my variable Java is not configured really well, but I have checked it is properly.

if its only about Java path then yes its not correct, if you see your PATH variable you can see there is one more value which is `C:/....JAVA` and you set your own path too. SO do one thing, remove any value from `PATH` variable witch lead to JAVA. just make sure there is only one `JAVA_HOME` in system variable and which points upto `bin` dir not bin and then in `PATH` variable add `JAVA_HOME\bin` try this — Dupinder Singh, May 25 '20 at 17:51
Is true, I have two JAVA_HOME variable (system and User). I deleted path JAVA_HOME in my user account. But I have the same situación for PATH variable. Do you want to delete JAVA_HOME\bin in Path variable into user account yet? And this only is in system account — Manuel Gomez-Rico Ubeda, May 25 '20 at 21:18
Now from git-bash I execute java -v, don´t recognize it. But if I execute jupyter notebook from git bash, pyspark work really, with this code — Manuel Gomez-Rico Ubeda, May 25 '20 at 21:36

score 0 · Answer 1 · answered May 25 '20 at 21:37

0

But if I execute jupyter notebook from git bash, pyspark work really, with this code

enter image description here

answered May 25 '20 at 21:37

Manuel Gomez-Rico Ubeda

1
1

score 0 · Answer 2 · answered May 29 '20 at 01:47

It's like the version mismatch causes this. As '_PYSPARK_DRIVER_CALLBACK_HOST' has been removed during version 0.10.7(here) to 2.3, and gets back from version 2.3.1. So, you should check your Spark version, as the SPARK_HOME points to the right one(at least 2.3.1).

from here

score 0 · Answer 3 · answered Jun 05 '20 at 09:28

I solved it when I put into the .bash_profile file the following line:

export path_java (in my pc)

In this way, when I execute pytest in bash console I pass testing without problems

Although from VS Code I don't pass test with debugger, I can continue with my python library

Error with pyspark in local when I execute pytest in VS Code from Git Bash

3 Answers3