How to define partitions to Dataframe in pyspark?

Asked May 13 '18 at 07:45

Active May 13 '18 at 10:43

Viewed 262 times

Suppose I read a parquet file as a Dataframe in pyspark, how can I specify how many partitions it must be?

I read the parquet file like this -

df = sqlContext.read.format('parquet').load('/path/to/file')

How may I specify the number of partitions to be used?

edited May 13 '18 at 10:43

asked May 13 '18 at 07:45

Ani Menon

1

Possible duplicate of [How to change partition size in Spark SQL](https://stackoverflow.com/questions/38249624/how-to-change-partition-size-in-spark-sql) – Alper t. Turker May 13 '18 at 08:23
@user9613318 Its not a duplicate of [that question](https://stackoverflow.com/questions/38249624/how-to-change-partition-size-in-spark-sql) about _partition size_. – Ani Menon May 13 '18 at 10:42

0 Answers0