7

What is the difference between Tez engine and Map Reduce engine in Hive and in which process which engine is better to use (for eg:joins, aggregation?)

Community
  • 1
  • 1
Bitanshu Das
  • 627
  • 2
  • 8
  • 21

3 Answers3

11

Tez is a DAG (Directed acyclic graph) architecture. A typical Map reduce job has following steps:

  1. Read data from file -->one disk access

  2. Run mappers

  3. Write map output --> second disk access

  4. Run shuffle and sort --> read map output, third disk access

  5. write shuffle and sort --> write sorted data for reducers --> fourth disk access

  6. Run reducers which reads sorted data --> fifth disk output

  7. Write reducers output -->sixth disk access

Tez works very similar to Spark (Tez was created by Hortonworks well before Spark):

  1. Execute the plan but no need to read data from disk.

  2. Once ready to do some calculations (similar to actions in spark), get the data from disk and perform all steps and produce output.

Only one read and one write.

The efficiency is increased by not going to disk multiple times. Intermediate results are stored in memory (not written to disks)

Jeevitesh Pola
  • 126
  • 1
  • 4
  • That said, imagine I'm going to run a Hive query that involves a join: if I use MR engine, then the steps you mentioned are going to be doubled while the pipeline remains only one when using Tez? – Memphis Meng Jul 09 '22 at 20:33
4

Tez is a DAG-based system, it's aware of all opération in such a way that it optimizes these operations before starting execution.

MapReduce model simply states that any computation can be performed by two kinds of computation steps – a map step and a reduce step. One pair of map and reduce does one level of aggregation over the data. Complex computations typically require multiple such steps.

Tez is usually running under MaprReduce, so it's just a MapReduce optimized with less and compacted steps.

Radhwane Chebaane
  • 839
  • 8
  • 23
2

Apache Tez is plug-in compatible with MapReduce but reduces the amount of disk access. Tez is always better than MapReduce.

However, there are also systems better than Hive + Tez, such as Spark SQL.

John Rotenstein
  • 241,921
  • 22
  • 380
  • 470
  • 1
    So for Joins and aggregation , which is superior? I was using Mapreduce and then shifted to Tez, but found that Tez was taking more time compared to mapreduce in case of multiple joins. – Bitanshu Das Jan 13 '17 at 10:20