How can we modify PySpark configuration on Jupyter

Question

I am currently working on a Jupyter (Lab) and PySpark 2.1.1.

I want to change spark.yarn.queue and master from a notebook. Because of the kernel spark and sc are available when I open a notebook.

Following this question, I tried

spark.conf.set("spark.yarn.queue", "my_queue")

But according to spark.sparkContext.getConf() the above line has no affect.

spark.conf.setMaster("yarn-cluster")

is not working, because there is no such a method for spark.conf.

Question: How can I change the configuration (queue and master) from a Jupyter notebook?

(Or should I set any environment variables?)

score 1 · Accepted Answer · answered Sep 05 '18 at 12:41

1

You can try to initialize spark beforehand, not in the notebook. Run this on your terminal:

export PYSPARK_DRIVER_PYTHON=jupyter
export PYSPARK_DRIVER_PYTHON_OPTS='notebook'

pyspark --master <your master> --conf <your configuration> <or any other option that pyspark supports>.

My source

answered Sep 05 '18 at 12:41

martinarroyo

9,389
3
38
75

The commands you wrote must work, but doesn't. I am looking for the reason. – H. Shindoh Sep 05 '18 at 13:55
I think that our big data team has configured in a wrong way (or not yet). Thank you for your help. – H. Shindoh Sep 05 '18 at 14:39
Sure. Let me know if I can give of any further assistance. – martinarroyo Sep 05 '18 at 15:01

How can we modify PySpark configuration on Jupyter

1 Answers1