2

I am having couple of Spark jobs which processes thousands of files every day. File size may very from MBs to GBs. After finishing job I usually save using the following code

finalJavaRDD.saveAsParquetFile("/path/in/hdfs"); OR
dataFrame.write.format("orc").save("/path/in/hdfs") //storing as ORC file as of Spark 1.4

Spark job creates plenty of small part files in final output directory. As far as I understand Spark creates part file for each partition/task - is this correct? How do we control amount of part files Spark creates?

Finally, I would like to create Hive table using these parquet/orc directory and I heard Hive is slow when we have large no of small files.

halfer
  • 19,824
  • 17
  • 99
  • 186
Umesh K
  • 13,436
  • 25
  • 87
  • 129

2 Answers2

5

You may want to try using the DataFrame.coalesce method to decrease the number of partitions; it returns a DataFrame with the specified number of partitions (each of which becomes a file on insertion).

To increase or decrease the partitions you can use Dataframe.repartition function. But coalesce does not cause shuffle while repartition does.

Community
  • 1
  • 1
zweiterlinde
  • 14,557
  • 2
  • 27
  • 32
0

Since 1.6 you can use repartition on data frame, which means you'll get 1 file per hive partition. Beware of large shuffles though, best to have your DF partitioned properly from starts if possible. See https://stackoverflow.com/a/32920122/2204206

Community
  • 1
  • 1
Lior Chaga
  • 1,424
  • 2
  • 21
  • 35