passing JSON argument as a string to python hadoop streaming application

Question

I want to pass a JSON string as command line argument to my reducer.py file but I'm unable to do so.

Command I execute is:

hadoop jar contrib/streaming/hadoop-streaming.jar -file /home/hadoop/mapper.py -mapper 'mapper.py' -file /home/hadoop/reducer.py -reducer 'reducer.py {"abc":"123"}' -input /user/abc.txt -output /user/output/

When I print argv array in reducer.py, it shows output as:

['/mnt/var/lib/hadoop/tmp/nm-local-dir/usercache/hadoop/appcache/application_1423459215008_0057/container_1423459215008_0057_01_000004/./reducer.py', '{', 'abc', ':', '123', '}']

First argument is the path of reducer.py but my second argument gets split by double quotes.

I want to achieve second argument as a complete JSON string. For example like: ['/mnt/var/lib/hadoop/tmp/nm-local-dir/usercache/hadoop/appcache/application_1423459215008_0057/container_1423459215008_0057_01_000004/./reducer.py','{"abc":"123"}']

So that I can load that argument as Json in reducer.py

Any help is appreciated. Thanks !

EDIT: Tried escaping JSON using command:

hadoop jar contrib/streaming/hadoop-streaming.jar -file /home/hadoop/mapper.py -mapper 'mapper.py' -file /home/hadoop/reducer.py -reducer 'reducer.py "{\"abc\":\"123\"}"' -input /user/abc.txt -output /user/output/

Gives output as:

['/mnt/var/lib/hadoop/tmp/nm-local-dir/usercache/hadoop/appcache/application_1423459215008_0058/container_1423459215008_0058_01_000004/./redu.py', '{\\', 'abc\\', ':\\', '123\\', '}']

score 1 · Accepted Answer · edited May 23 '17 at 12:06

1

You need to put your json inside double quotes with proper escaping: "{\"abc\":\"123\"}" but chances are that your input will be processed Hadoop before being passed to your script.

If this doesn't work you can try passing your arguments via environment with -cmdenv name=value. See How do I pass a parameter to a python Hadoop streaming job? for more details.

edited May 23 '17 at 12:06

Community

1
1

answered Feb 11 '15 at 06:49

Saulius Žemaitaitis

2,956
3
29
36

1

This might not work. Take a look at [this](http://stackoverflow.com/questions/17438852/pass-in-string-as-argument-without-it-being-treated-as-raw) – khajvah Feb 11 '15 at 06:58
@frgtn Sadly it didnt work. It gives output as `['{\\', 'abc\\', ':\\', '123\\', '}']` – shahsank3t Feb 11 '15 at 07:06
@shahsank3t can you update your question with full escaped input and output? – Saulius Žemaitaitis Feb 11 '15 at 07:11
@frgtn I have noticed some weird thing after testing. It works if I do `'{\"gme\"}'` but it doesn't work if I do `"{\"gme\"}"` – khajvah Feb 11 '15 at 07:18
@khajvah I've tested this by invoking script via shell. Hadoop may be processing the commands. – Saulius Žemaitaitis Feb 11 '15 at 07:23
Sorry for late reply, I've used `-cmdenv name=value` and it worked for me. Thanks guys. – shahsank3t Feb 16 '15 at 09:13

passing JSON argument as a string to python hadoop streaming application

1 Answers1