Number of files in Mapper job without Reducer

Question

I set Reducer to 0, running only Mapper job. Lets say 10 Nodes are executing the Mapper job. I understand 10 Mappers produces 10 files in HDFS. But, how many number of files would be produced as in final output for the result?

No.say if reducers enabled and despite number of reducers executed the aggregation, the result file will be one. So my question is about how many number of files without reducers? — Sats, Mar 25 '18 at 14:41
But reducers aren't enabled... You said you set it to zero. You also said 10 mappers makes 10 files, which is correct, so what is your actual question? — OneCricketeer, Mar 25 '18 at 19:22

score 0 · Accepted Answer · answered Mar 24 '18 at 19:20

0

A job without reducers is a Map-only job, every mapper will produce an output file.

Please check this answer out: `

When you have a map-only task, there is not shuffling at all, which means that mappers will write the final output directly to the HDFS.

answered Mar 24 '18 at 19:20

dbustosp

4,208
25
46

So will there be multiple files for the user to check from hdfs? – Sats Mar 25 '18 at 14:39
@Sathish Yes. Unless you have 1reduce, that way you will have to check only one file. However, that reduce potentially will become a bottleneck. – dbustosp Mar 25 '18 at 15:18
Agree. So, if 5 reducers enabled, then the end result file will be one, isn't it? – Sats Mar 26 '18 at 15:45
If 5 reducers enabled, then you will have 5 files as an output. – dbustosp Mar 26 '18 at 16:29
Grand. Thank you ! – Sats Mar 27 '18 at 16:48

Number of files in Mapper job without Reducer

1 Answers1