Write to multiple outputs by key Scalding Hadoop, one MapReduce Job

Question

How can you write to multiple outputs dependent on the key using Scalding(/cascading) in a single Map Reduce Job. I could of course use .filter for all the possible keys, but that is a horrible hack, which will fire up many jobs.

score 6 · Accepted Answer · edited May 19 '18 at 01:31

6

There is TemplatedTsv in Scalding (from version 0.9.0rc16 and up), exactly same as Cascading TemplateTsv.

Tsv(args("input"), ('COUNTRY, 'GDP))
.read
.write(TemplatedTsv(args("output"), "%s", 'COUNTRY))
// it will create a directory for each country under "output" path in Hadoop mode.

edited May 19 '18 at 01:31

proteus

467
4
21

answered Jun 25 '14 at 12:04

morazow

251
1
8

This looks even more flexible than what I requested!! Thanks. Could you say from which Scalding version this is? Is it 0.10.0 and above? or 0.9.0? – samthebest Jun 26 '14 at 14:26
1

From codebase, it looks to be available from 0.9.0rc16 version. – morazow Jun 27 '14 at 12:18
@morazow is there any way to drop the fields used in the template string? In your example, basically I want the resulting files to have only the 'GDP' field in the resulting output. – Syllepsis Oct 14 '14 at 13:48
@Syllepsis That is really good question; but, I do not know if it is possible with current TemplatedTsv implementation. However, you can make another, your own, MyTemplatedTsv like here https://github.com/twitter/scalding/blob/0.11.0/scalding-core/src/main/scala/com/twitter/scalding/TemplateSource.scala#L99-L105 and add "override val fields = Fields.ALL" and specify the fields to be written when calling that tap. Could you please reply here if you test that? – morazow Oct 15 '14 at 13:01

score 0 · Answer 2 · edited May 23 '17 at 12:02

0

Use MultipleOutputFormat and extrapolate from these other SO questions to write a custom output class using the output format: Create Scalding Source like TextLine that combines multiple files into single mappers, Compress Output Scalding / Cascading TsvCompressed

edited May 23 '17 at 12:02

Community

1
1

answered Jun 02 '14 at 12:47

samthebest

30,803
25
102
142

score 0 · Answer 3 · answered Jun 02 '14 at 18:27

0

This suggestion on the Cascading User group suggests to use Cascading TemplateTap. Not sure how to connect this to Scalding though.

answered Jun 02 '14 at 18:27

Sasha O

3,710
2
35
45

That certainly looks promising, care to provide Scalding code for people's copy and paste needs? :) – samthebest Jun 02 '14 at 20:56

Write to multiple outputs by key Scalding Hadoop, one MapReduce Job

3 Answers3

Linked