1

I am trying to figure out how to develop apache-spark program in PyCharm. I have followed article in this link. I define SPARK_HOME and add pyspark to Python path well. There is no error in importing pyspark modules and autocomplete works fine.

However I get an error on defining SparkContext when I run the program in PyCharm.

Error: Must specify a primary resource (JAR or Python or R file)
Run with --help for usage help or --verbose for debug output
...
...
Exception: Java gateway process exited before sending the driver its port number

I managed to run the program on terminal with submit-spark. Do I need to change the configuration on PyCharm or Is there anyway to run submit-spark instead of python in PyCharm?

endeavour90
  • 87
  • 3
  • 7
  • this should help you https://medium.com/@gauravmshah/pyspark-on-intellij-with-packages-auto-complete-5e3208504707 – Gaurav Shah Dec 13 '18 at 18:23

1 Answers1

0

If you are fine by terminal submit-spark you can add a run configuration that does that for you. Otherwise you can see some configuration in Edit Run/Debug Configurations window as well. This post in particular can you get you there.

Serzhan Akhmetov
  • 2,898
  • 2
  • 13
  • 26
mehmetminanc
  • 1,359
  • 9
  • 14