3

I would like to know if the order of the data records matter (performance wise) when joining two tables?
P.S. I am not using any map-side join or bucket join.

Thank you!

leftjoin
  • 36,950
  • 8
  • 57
  • 116
James
  • 57
  • 5

1 Answers1

3

On the one hand order should not matter because during shuffle join files are being read by mappers in parallel, also files may be splitted between few mappers or vice-versa, one mapper can read few files, then mappers output passed to each reducer. And even if data was sorted it is being read and distributed not in it's order due to parallelism.

On the other hand, sorting improves compression depending on the data entropy. Similar data can be compressed better. Therefore files ordered compressed are smaller and they will be read faster during join query execution. This may improve join speed because mappers will read data faster and internal indexes in ORC work efficiently if data was sorted by filter columns during load and PPD is enabled. Sorted and compressed file size can be reduced x3 times or even more, it will result in x3 less mappers.

Sorting is efficient when you are writing and sorting once and reading many times.

leftjoin
  • 36,950
  • 8
  • 57
  • 116
  • Hi leftjoin, thank you for your input. May I ask if I do not use any compression, ordering the data does not affect/improve the join performance right? – James Nov 23 '17 at 02:46
  • 1
    @James Sorting may improve in some cases if you are using ORC and sort+distribute before insert into ORC table: https://community.hortonworks.com/articles/75501/orc-creation-best-practices.html – leftjoin Nov 23 '17 at 07:47