1

How many number of partitions will be there if I have 100 gb of data in spar? If someone says we have 20 will that be enough for the memory

Jacek Laskowski
  • 72,696
  • 27
  • 242
  • 420
prateek kumar
  • 11
  • 1
  • Hi Prateek, wellcome to SO. Please, read this link. https://stackoverflow.com/questions/42171499/get-current-number-of-partitions-of-a-dataframe – aironman Sep 05 '20 at 19:10

1 Answers1

0

The number of partitions depends on the data source in use. For file-based data sources that use HDFS as a storage there will be as many partitions as there are blocks. Other data sources can report this requirement differently (based on their own internal logic).

Jacek Laskowski
  • 72,696
  • 27
  • 242
  • 420