0

Q1. Will adhoc (dynamic) repartition of the data a line before a join help to avoid shuffling or will the shuffling happen anyway at the repartition and there is no way to escape it?

Q2. should I repartition/partitionBy/bucketBy? what is the right approach if I will join according to column day and user_id in the future? (I am saving the results as hive tables with .write.saveAsTable). I guess to partition by day and bucket by user_id but that seems to create thousands of files (see Why is Spark saveAsTable with bucketBy creating thousands of files?)

Nikunj Kakadiya
  • 2,689
  • 2
  • 20
  • 35
Hanan Shteingart
  • 8,480
  • 10
  • 53
  • 66
  • @Hanan_Shteingart For Q1 there is no guarantee that doing repartition before join would eliminate shuffle .It would definitely shuffle the data unless there is small dataframe that can be broadcasted. – Nikunj Kakadiya Jan 28 '21 at 07:25

1 Answers1

0

Some 'guidance' off the top of my head, noting that title and body of text differ to a degree:

Question 1:

  • A JOIN will do any (hash) partitioning / repartitioning required automatically - if needed and if not using a Broadcast JOIN. You may set the number of partitions for shuffling or use the default - 200. There are more parties (DF's) to consider.

  • repartition is a transformation, so any up-front repartition may not be executed at all due to Catalyst optimization - see the physical plan generated from the .explain. That's the deal with lazy evaluation - determining if something is necessary upon Action invocation.

Question 2:

  • If you have a use case to JOIN certain input / output regularly, then using Spark's bucketBy is a good approach. It obviates shuffling. The databricks docs show this clearly.

  • A Spark schema using bucketBy is NOT compatible with Hive. so these remain Spark only tables, unless this changed recently.

  • Using Hive partitioning as you state depend on push-down logic, partition pruning etc. It should work as well but you may have have different number of partitions inside Spark framework after the read. It's a bit more complicated than saying I have N partitions so I will get N partitions on the initial read.

thebluephantom
  • 16,458
  • 8
  • 40
  • 83