3

I am facing a weird behavior with the fabric2 command module. Those commands work like a charm: 

connect = Connection(host=h, user=u ,connect_kwargs={"password":p})
connect.run('mkdir temp_streaming')
connect.put(m, "temp_streaming/mapper.py")
connect.put(r, "temp_streaming/reducer.py")

But when it comes to this one:

input_path = input("Please choose the input of your program (on your HDFS File System) ")
output_path = input("Please choose a name for your output folder ")

main_command = str('yarn jar '+ jar_path+' -files mapper.py,reducer.py -mapper temp_streaming/mapper.py -reducer temp_streaming/reducer.py -input '+ input_path + ' -output '+ output_path)

connect.run(main_command)

I get this error I can't figure out:

bash: yarn: command not found
Traceback (most recent call last):
  File "__main.py__", line 77, in <module>
    main()
  File "__main.py__", line 65, in main
   RunMapReduce(mapper, reducer, jar_path)
  File "__main.py__", line 46, in RunMapReduce
   connect.run(main_command)
 File "<decorator-gen-3>", line 2, in run
 File "/usr/local/lib/python3.5/dist-packages/fabric2/connection.py", line 30, in opens
return method(self, *args, **kwargs)
 File "/usr/local/lib/python3.5/dist-packages/fabric2/connection.py", line 586, in run
return self._run(runner, command, **kwargs)
File "/usr/local/lib/python3.5/dist-packages/invoke/context.py", line 100, in _run
return runner.run(command, **kwargs)
File "/usr/local/lib/python3.5/dist-packages/invoke/runners.py", line 268, in run
return self._run_body(command, **kwargs)
File "/usr/local/lib/python3.5/dist-packages/invoke/runners.py", line 401, in _run_body
raise UnexpectedExit(result)
invoke.exceptions.UnexpectedExit: Encountered a bad command exit code!

Command: 'yarn jar /home/hadoop/hadoop/share/hadoop/tools/lib/hadoop-streaming-2.8.4.jar -files mapper.py,reducer.py -mapper temp_streaming/mapper.py -reducer temp_streaming/reducer.py -input /books/kafka_metamorphosis.txt -output /test'

Exit code: 127

Stdout: already printed

Stderr: already printed

It acts as if the distant server did not know the yarn command even though this bash command works when I directly log on the server with ssh and run it. It seems to me that it is related with this library but I couldn't solve my problem with the documentation.

Arount
  • 9,853
  • 1
  • 30
  • 43
zar3bski
  • 2,773
  • 7
  • 25
  • 58

2 Answers2

3

I faced this problem too : turns out that the SSH run by Fabric is a non-interactive shell. The $PATH variable which contains the path to you yarn bin is set in /etc/profile or ~/.bash_profile, or ~/.bashrc, none of which get sourced.

Here is the link, along with some solutions.

Personally, I am not a fan of playing with those files => What I just did (prior to stumbling upon this link) is I used the full paths of the binaries : /usr/bin/yarn [the rest of your command]

Ismaïl Mourtada
  • 452
  • 5
  • 11
0

I somehow managed to get it to work with the keyword argument warn=True like so:

result = connection.run(command, warn=True)

This option was not obvious on the documentation, needed to search an upgrade guide to find it. Although this seems not to be an issue, I would strongly suggest an improvement in Fabric2 documentation, as this is I reckon an immensely likely scenario.

ChrisGPT was on strike
  • 127,765
  • 105
  • 273
  • 257
Naman Madharia
  • 1
  • 2