How can I embed a Python program in a bash command if it needs both a loop and an import?

Question

I'm trying to use Python to extract info from some JSON (on a system where I can't install jq). My current approach runs afoul of the syntax restrictions described in Why can't use semi-colon before for loop in Python?. How can I modify this code to still work in light of this limitation?

My current code looks like the following:

$ SHIFT=$(aws ec2 describe-images --region "$REGION" --filters "Name=tag:Release,Values=$RELEASE_CODE_1.2003.2")
$ echo "$SHIFT" | python -c "import sys, json; for image in json.load(sys.stdin)['Images']: print image['ImageId'];"
  File "<string>", line 1
    import sys, json; for image in json.load(sys.stdin)['Images']: print image['ImageId'];
                        ^

SyntaxError: invalid syntax

Since Python's syntax doesn't allow a for loop to be separated from a prior command with a semicolon, how can I work around this limitation?

BTW, personally, I'd use `jq -r '.Images[].ImageId' <<<"$SHIFT"` rather than embedding Python here. — Charles Duffy, Jan 18 '18 at 21:55
@CharlesDuffy: File "", line 1 import sys, json;\nfor img in json.load(sys.stdin)['Images']:\n\tprint img['ImageId'] ^ SyntaxError: unexpected character after line continuation character — iam-decoder, Jan 18 '18 at 21:57
Hmm. Let me see about editing this to make it less duplicative so it can be re-opened -- there probably *is* room to find a distinct question in here. — Charles Duffy, Jan 18 '18 at 22:07
@vaultah, I've tried to edit this into a distinct question, since an answer that talked about bash syntax that can compress multi-line strings into a one-liner isn't really topical on the other question, yet is applicable to the needs of the OP here. Do you agree that this is now distinct and useful? — Charles Duffy, Jan 18 '18 at 22:12

Charles Duffy · Accepted Answer · 2018-01-19T17:43:56.207

There are several options here:

Pass your code as a multi-line string. Note that " is used to delimit Python strings rather than the original ' here for the sake of simplicity: A POSIX-compatible mechanism to embed a literal ' in a single-quoted string is possible, but quite ugly.
```
extractImageIds() {
  python -c '
import sys, json
for image in json.load(sys.stdin)["Images"]:
    print image["ImageId"]
' "$@"
}
```
Use bash's C-style escaped string syntax ($'') to embed newlines, as with $'\n'. Note that the leading $ is critical, and that this doesn't work with /bin/sh. See the bash-hackers' wiki on ANSI C-like strings for details.
```
extractImageIds() { python -c $'import sys, json\nfor image in json.load(sys.stdin)["Images"]:\n\tprint image["ImageId"]' "$@"; }
```

Use __import__() to avoid the need for a separate import command.

extractImageIds() { python -c 'for image in __import__("json").load(__import__("sys").stdin)["Images"]: print image["ImageId"]' "$@"; }

Pass the code on stdin and move the input onto argv; note that this only works if the input doesn't overwhelm your operating system's allowed maximum command-line size. Consider the following example:
```
extractImageIds() {
  # capture function's input to a variable
  local input=$(</dev/stdin) || return
  # ...and expand that variable on the Python interpreter's command line
  python - "$input" "$@" <<'EOF'
import sys, json
for image in json.loads(sys.argv[1])["Images"]:
    print image["ImageId"]
EOF
}
```
Note that $(</dev/stdin) is a more efficient bash-only alternative to $(cat); due to shell builtin support, it works even on operating systems where /dev/stdin doesn't exist as a file.

All of these have been tested as follows:

extractImageIds <<<'{"Images": [{"ImageId": "one"}, {"ImageId": "two"}]}'

To efficiently provide stdin from a variable, one could run extractImageIds <<<"$variable" instead. Note that the "$@" elements in the wrapper are there to ensure that sys.argv is populated with arguments to the shell function -- where sys.argv isn't referenced by the Python code being run, this syntax is optional.

How can I embed a Python program in a bash command if it needs both a loop and an import?

1 Answers1