I am trying to run the following command via paramiko exec_command. The stdout is different when I run this command via paramiko vs just pasting it into the shell in an ssh terminal session (I printed the command from the python script and then pasted that output directly to the ssh terminal).
The command is to submit jobs to an LSF queue based on when other jobs are completed, but I am not sure that my issue has anything to do with that:
bsub -w "done(100001) && done(100002) && done(100003)" < path/to/script.sh
when the script is run via paramiko, this is the code and stdout: in:
alljobs=$(cat path/to/job_list.txt | sed '$!{:a;N;s/\n/\) \&\& done\(/;ta}')
echo $alljobs
cmd='bsub -w "done('$alljobs')" < path/to/script1.sh'
echo $cmd
var="$(echo ${cmd}|sh)"
echo $var
var0=$(echo $var | awk -F "[\<\>]" '{print $2}')
echo $var0
var1="$(bsub -w "done($var0)" < path/to/script2.sh)"
echo $var1
echo "this is the end of the script"
out:
100001) && done(100002) && done(100003
bsub -w "done(100001) && done(100002) && done(100003)" < path/to/script1.sh
this is the end of the script
However, if I just log in to an ssh session and paste in bsub -w "done(100001) && done(100002) && done(100003)" < path/to/script.sh
and this is in in/out:
in:
alljobs=$(cat path/to/job_list.txt | sed '$!{:a;N;s/\n/\) \&\& done\(/;ta}')
echo $alljobs
cmd='bsub -w "done('$alljobs')" < path/to/script1.sh'
echo $cmd
var="$(echo ${cmd}|sh)"
echo $var
var0=$(echo $var | awk -F "[\<\>]" '{print $2}')
echo $var0
var1="$(bsub -w "done($var0)" < path/to/script2.sh)"
echo $var1
echo "this is the end of the script"
out:
100001) && done(100002) && done(100003
bsub -w "done(100001) && done(100002) && done(100003)" < path/to/script1.sh
Job <111111> is submitted to queue <premium>.
111111
Job <222222> is submitted to queue <premium>.
this is the end of the script
I am guessing that this is an issue with paramiko vs an interactive session, but I am not certain.
Edit: tried adding get_pty=True (and also False) to exec_command, didn't make a difference.
Edit #2: Printed $PATH in the shell and from paramiko, they were indeed different even with get_pty=True, with the one used when I log in manually having my username in the $PATH, which was not the case with the $PATH used by paramiko. So that's probably the problem, now to see if I can do anything about it.
Edit #3: This does not sound encouraging, or at least it sounds like I might need help from the sysadmins.
Edit #4: I tried invoking the shell script using ssh.invoke_shell and writing the command directly to stdin. Although that did invoke the script, the results were the same. But then I checked the $PATH and saw that I was indeed using the same $PATH as when I log in manually under invoke_shell. So I perhaps $PATH was not the problem?
My test use of invoke_shell was this:
cmd = "echo $PATH > " + full_path + "/printpath.txt\n"
print(cmd)
channel = ssh.invoke_shell()
stdin = channel.makefile('wb')
stdout = channel.makefile('rb')
stdin.write(cmd)
print(stdout.read())
Edit #5: The solution!
It turns out that I needed to use a combination of things to get it working. One was running the command using bash --login (as suggested by @cxw), and then I also had to modify my command to triple-escape the inner quotation marks. So my working python was:
cmd = "bsub -w \\\"" + jobz + "\\\" < " + exp_path + "/bsubmit2_" + EXPNAME + ".sh"
explicit = 'bash --login -c "' + cmd + '"'
print(explicit)
stdin, stdout, stderr = ssh.exec_command(explicit)
