How to convert shell script in one line and execute it from Python subprocess?

Question

I have my below simple shell script which I am trying to convert in one single line so that I can execute it from my Python program using subprocess -

#!/bin/bash
set -e

COUNT=60   #number of 10 second timeouts in 10 minutes
SUM_SYNCS=0
SUM_SYNCS_BEHIND=0
HOSTNAME=$hostname

echo $HOSTNAME

while [[ $COUNT -ge "0" ]]; do

#send the request, put response in variable
DATA=$(wget -O - -q -t 1 http://$HOSTNAME:8080/beat)

#grep $DATA for syncs and syncs_behind
SYNCS=$(echo $DATA | grep -oE 'num_syncs: [0-9]+' | awk '{print $2}')
SYNCS_BEHIND=$(echo $DATA | grep -oE 'num_syncs_behind: [0-9]+' | awk '{print $2}')

echo $SYNCS
echo $SYNCS_BEHIND

#verify conditionals
if [[ $SYNCS -gt "8" && $SYNCS_BEHIND -eq "0" ]]; then exit 0; fi

#decrement the counter
let COUNT-=1

#wait another 10 seconds
sleep 10

done

I converted the above script in one line like this -

jsonStr = {"script": "#!/bin/bash\nset -e\n\nCOUNT=60   #number of 10 second timeouts in 10 minutes\nSUM_SYNCS=0\nSUM_SYNCS_BEHIND=0\nHOSTNAME=$hostname\t\n\necho $HOSTNAME\n\nwhile [[ $COUNT -ge \"0\" ]]; do\n\n#send the request, put response in variable\nDATA=$(wget -O - -q -t 1 http://$HOSTNAME:8080/beat)\n\n#grep $DATA for syncs and syncs_behind\nSYNCS=$(echo $DATA | grep -oE 'num_syncs: [0-9]+' | awk '{print $2}')\nSYNCS_BEHIND=$(echo $DATA | grep -oE 'num_syncs_behind: [0-9]+' | awk '{print $2}')\n\necho $SYNCS\necho $SYNCS_BEHIND\n\n#verify conditionals\nif [[ $SYNCS -gt \"8\" && $SYNCS_BEHIND -eq \"0\" ]]; then exit 0; fi\n\n#decrement the counter\nlet COUNT-=1\n\n#wait another 10 seconds\nsleep 10\n\ndone\n"}

And I am trying to execute this using Python subprocess but everytime I am getting an error -

  File "C:\Python27\lib\json\__init__.py", line 326, in loads
    return _default_decoder.decode(s)
  File "C:\Python27\lib\json\decoder.py", line 366, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
TypeError: expected string or buffer

Below is my full Python code which I am trying out -

jsonStr = {"script": "#!/bin/bash\nset -e\n\nCOUNT=60   #number of 10 second timeouts in 10 minutes\nSUM_SYNCS=0\nSUM_SYNCS_BEHIND=0\nHOSTNAME=$hostname\t#\n\necho $HOSTNAME\n\nwhile [[ $COUNT -ge \"0\" ]]; do\n\n#send the request, put response in variable\nDATA=$(wget -O - -q -t 1 http://$HOSTNAME:8080/beat)\n\n#grep $DATA for syncs and syncs_behind\nSYNCS=$(echo $DATA | grep -oE 'num_syncs: [0-9]+' | awk '{print $2}')\nSYNCS_BEHIND=$(echo $DATA | grep -oE 'num_syncs_behind: [0-9]+' | awk '{print $2}')\n\necho $SYNCS\necho $SYNCS_BEHIND\n\n#verify conditionals\nif [[ $SYNCS -gt \"8\" && $SYNCS_BEHIND -eq \"0\" ]]; then exit 0; fi\n\n#decrement the counter\nlet COUNT-=1\n\n#wait another 10 seconds\nsleep 10\n\ndone\n"}

j = json.loads(jsonStr)

print "start"
proc = subprocess.Popen(j['script'], shell=True, stdout=subprocess.PIPE, stderr=subprocess.PIPE, executable='/bin/bash')
(stdout, stderr) = proc.communicate()
print stdout
print stderr
print "end"

What is wrong in my shell script which I have converted in one line? Any thoughts?

NOTE: All my other shell scripts are working which are also in one liner from the same Python subprocess call, only this is having some problem.

Why aren't you using a [triple quoted string](http://docs.python.org/release/1.4/tut/node70.html)? — that other guy, Mar 14 '14 at 00:23
possible duplicate of [Running bash script from within python](http://stackoverflow.com/questions/13745648/running-bash-script-from-within-python) — aruisdante, Mar 14 '14 at 00:25
There's also no need to load it in like this, you can simply execute it directly. See my flagged duplicate answer. — aruisdante, Mar 14 '14 at 00:26
Somehow all my other scripts are working like this way only in Python subprocess call so I don't want to change that call the way I am doing currently.. Because earlier, I tried with the way you have suggested, so some scripts were running fine and some were not.. — arsenal, Mar 14 '14 at 00:28

score 2 · Accepted Answer · answered Mar 14 '14 at 00:25

2

The error you are getting is from json.loads(jsonStr). json.loads require a str or unicode not a dict. jsonStr is already a dict you don't need json.loads

answered Mar 14 '14 at 00:25

Aamir Rind

38,793
23
126
164

I see.. So If I am doing like this - `proc = subprocess.Popen(jsonstr['script'], shell=True, stdout=subprocess.PIPE, stderr=subprocess.PIPE, executable='/bin/bash')` then it should work fine right? I tried this way and it doesn't work as well. error = `string indices must be integers, not str` – arsenal Mar 14 '14 at 00:30
@Webby Remove this line `j = json.loads(jsonStr)` and try with `jsonStr['script']` – Aamir Rind Mar 14 '14 at 00:35
I already tried that and this is the error I got `TypeError: string indices must be integers, not str` – arsenal Mar 14 '14 at 00:36
Using bash syntax in a Python data structure will usually fail. Some simple constructs like strings may use a similar-enough syntax to represent the same value in both, but in general will not. For example, bash uses square brackets for purposes other than indexing an iterable (which is probably the cause of your most recent error). Either capture the bash script as a literal string (the previous triple-quote suggestion, and don't forget the leading `r`), or capture the script as a file. Trying to use bash syntax in Python is pointless and foolish. – Chris Johnson Mar 14 '14 at 00:52

How to convert shell script in one line and execute it from Python subprocess?

1 Answers1