1

I am running zeppelin with Spark on yarn. Option --py-files(SPARK_SUBMIT_OPTIONS) does not work in zeppelin. Is there any alternative to --py-files in zeppelin.

NOTE: I can upload files using option: --files but then it does not add those files in PYTHONPATH. Hence I need an alternative to --py-files in zeppelin.

Santosh Kumar
  • 761
  • 5
  • 28

1 Answers1

0

I'm not familiar with Zeppelin, but you can achieve the equivalent of --py-files by using the addPyFile method on the context (See https://spark.apache.org/docs/1.6.1/api/python/pyspark.html). HTH

Ali Tajeldin
  • 61
  • 4
  • I was able to get over the problem by replicating the python files to all nodes in the cluster and then updating PYTHONPATH accordingly while starting zeppelin. – Santosh Kumar Mar 29 '17 at 19:01
  • https://stackoverflow.com/questions/40808064/pyspark-append-executor-environment-variable – mathtick Feb 26 '18 at 19:04