How to only do quote removal in a shell?

Question

Given a string from an untrusted source, e.g.

MALICIOUS_INPUT="$(awk -F = '/^VERSION=/ {print $2}' /path/to/compromised/etc/os-release | head -n 1)"

is it possible to just apply pure shell quote removal (see Shell Command Language (IEEE Std 1003.1-2017) and Bash manual) to that string i.e. without doing variable expansions, arithmetic expansions, command substitution and similar?

This is needed, for example to parse strings from os-release files without source-ing the files.

Input	Expected result
`'\"'`	`\"`
`"\""`	`"`
`'$foo${foo}$(pwd)$((1+2))'`	`$foo${foo}$(pwd)$((1+2))`
`"$foo${foo}$(pwd)$((1+2))"`	`$foo${foo}$(pwd)$((1+2))`

The easy answer here is to drop to a non-bash language; Python's `shlex.split()` does only quote removing and word-splitting. — Charles Duffy, Mar 29 '21 at 17:54
`xargs` can also be leveraged to do _almost_, but not quite, shell-compatible parsing. (The corner cases are annoying, but if you run into them, you know the input is hostile -- no legitimate `/etc/os-release` will be containing multi-line strings or the like). Also, I'm quite sure I've already described the technique (as well as the use of Python's `shlex` module for the same purpose) in existing Q&A entries here. — Charles Duffy, Mar 29 '21 at 17:55
...here we are: [Reading quoted/escaped arguments correctly from a string](https://stackoverflow.com/a/31485948/14122) — Charles Duffy, Mar 29 '21 at 17:59

score 2 · Accepted Answer · answered Mar 29 '21 at 18:12

Comparing applicability of the preexisting answers on Reading quoted/escaped arguments correctly from a string to this question:

parse_with_xargs() {
  xargs printf '%s\0' <<<"$*"
}

parse_with_python() {
  python -c '
import shlex, sys
for item in shlex.split(sys.stdin.read()):
    sys.stdout.write(item + "\0")
' <<<"$*"
}

readarray -t example_lines <<'EOF'
'\"'
"\""
'$foo${foo}$(pwd)$((1+2))'
"$foo${foo}$(pwd)$((1+2))"  
EOF

for line in "${example_lines[@]}"; do
  printf 'Input line:         %s\n' "$line"
  printf 'Parsed with xargs:  '; parse_with_xargs "$line" 2>&1; echo
  printf 'Parsed with python: '; parse_with_python "$line" 2>&1; echo
  echo
done

Output:

Input line:         '\"'
Parsed with xargs:  \"
Parsed with python: \"

Input line:         "\""
Parsed with xargs:  xargs: unmatched double quote; by default quotes are special to xargs unless you use the -0 option

Parsed with python: "

Input line:         '$foo${foo}$(pwd)$((1+2))'
Parsed with xargs:  $foo${foo}$(pwd)$((1+2))
Parsed with python: $foo${foo}$(pwd)$((1+2))

Input line:         "$foo${foo}$(pwd)$((1+2))"  
Parsed with xargs:  $foo${foo}$(pwd)$((1+2))
Parsed with python: $foo${foo}$(pwd)$((1+2))

How to only do quote removal in a shell?

1 Answers1