Maybe this question is strongly related to How to set pivotMaxValues in pyspark?. But I haven't find any clear answer about this.
So, how I change spark.sql.pivotMaxValues in my code or configuration file? Where can I find the configuration file?
Asked
Active
Viewed 4,327 times
5
eliasah
- 39,588
- 11
- 124
- 154
Aries Fitriawan
- 51
- 1
- 4
-
7If your are using Spark 2.0+ you can do that `spark.conf.set('spark.sql.pivotMaxValues', u'50000')` where spark is your sparkSession – abdelhadi danba Aug 07 '18 at 13:43
-
This comment also works at the start of a pyspark notebook. – birdsarah Jan 11 '20 at 00:31