0

https://issues.apache.org/jira/browse/SPARK-832

Is this issue actually resolved? How can you include PYTHONPATH in spark workers in spark 2.2? Looking for someway that does not involve some global change (i.e. you are not allowed to use .profile, .bashrc or dist a change to every node on the cluster).

SPARK_HOME is where spark lives. Docs seems to indicate that this is also where conf/* are picked up. Is this correct?

UPDATE: I think the answer is here: Pyspark append executor environment variable

This really was not very well linked from the search results for PYTHONPATH. I have attempted to scatter breadcrumbs everywhere to help out.

mathtick
  • 6,487
  • 13
  • 56
  • 101
  • 1
    https://stackoverflow.com/questions/40808064/pyspark-append-executor-environment-variable – mathtick Feb 26 '18 at 19:05
  • With spark-submit its possible to set the PYTHON-PATH for the worker on the command line with --conf spark.pyspark.python=/usr/bin/python3 – Praveen Mar 30 '18 at 23:32

0 Answers0