bourne shell does unexpected word breaks

Question

Using MAC OS X and bourne shell:

I like to iterate through a list of items, for which some of the items include blanks. Each item should be treated as a whole, but they are splitted as shown below. If I "hard code" it like this, it is ok:

echo "hard coded :\n"
for i in 'a a a' 'bcd' 'e e'
do
 echo "$i"
done

But putting the same string into a variable, it is wrong:

echo "\nsecond loop:\n"
#
# str is normally coming from a comma substitution
str="'a a a' 'bcd' 'e e'"
echo $str
for i in  ${str}
do
 echo "$i"
done

The result is:

hard coded :
a a a
bcd
e e

second loop:

'a a a' 'bcd' 'e e'
'a
a
a'
'bcd'
'e
e'

MacOS doesn't have a Bourne shell available. (To be clear, `/bin/sh` on modern systems is POSIX sh, or a POSIX-superset shell such as bash or ksh, not Bourne). — Charles Duffy, Jan 06 '17 at 02:20
Also, this is *completely* expected behavior. See [BashFAQ #50](http://mywiki.wooledge.org/BashFAQ/050). In short: Literal quotes (quotes that are data) don't behave as syntactic quotes (quotes that are syntax) do. — Charles Duffy, Jan 06 '17 at 02:22

Adam Katz · Answer 1 · 2017-01-06T03:00:06.997

1

Here is some code that defines a CSV and then iterates over it without needing xargs or another external program. This is fully portable POSIX and will work in bash or any /bin/sh.

#!/bin/sh
str="a a a,*,e e"
echo $str
while [ "$str" != "${str#*,}" ]; do
  echo "${str%%,*}" # just the first CSV item
  str="${str#*,}"   # remove the first CSV item
done
echo "$str"         # the final (or only) CSV item

This uses shell variable parameter expansion.

Original answer

You could use a different delimiter and set $IFS (the input field separator) to key on it. I'm using a comma here because you've implied your original input uses comma-separated values:

#!/bin/sh -f

str="a a a,bcd,e e"
echo $str
OLDIFS="$IFS"
IFS=","
for i in  ${str}
do
 echo "$i"
done
IFS="$OLDIFS"

This produces:

a a a,bcd,e e
a a a
bcd
e e

The -f flag to the shell is important to suppress wildcard expansion ("globbing") in the event of an asterisk or similar character existing in the input.

This works in POSIX, Bourne, bash, korn, and most others in that family.

edited Jan 06 '17 at 03:00

answered Jan 06 '17 at 01:43

Adam Katz

14,455
5
68
83

I would expect, with `str="a a a,*,e e"`, the output to contain a literal `*`. Behavior that instead would emit a list of files in the current directory at least calls for a warning, no? – Charles Duffy Jan 06 '17 at 02:26
Try it. You should find that since the declaration is quoted, the wildcard expansion is suppressed and you will indeed get a literal `*`. If that weren't the case, you would have all kinds of ways of exploiting the code. If you actually _want_ the wildcard expansion, you'll have to use `eval` ... and then you have to perform lots of safety tests to ensure nothing malicious is going on. – Adam Katz Jan 06 '17 at 02:41
I did try it. I get a directory listing. – Charles Duffy Jan 06 '17 at 02:42
(This isn't as unsafe as `eval` -- it's just globbing as a side effect of unquoted expansion, which you could turn off with `set -f` -- but it *is* still an unintended side effect). – Charles Duffy Jan 06 '17 at 02:43
Well, there's a vulnerability for you. I'll suppress globbing in the shebang. – Adam Katz Jan 06 '17 at 02:44
1

@AdamKatz Pathname expansion does not occur during assignments anway: `foo=*` and `foo="*"` both assign the single-character string `*` to `foo`. Only when the unquoted parameter is *expanded* is the `*` subject to pathname expansion. – chepner Jan 06 '17 at 02:46
Yeah, I was worried about that. I was going to add another answer but you've already marked this as a duplicate to something that I'm not sure _is_ a duplicate. I'll instead revise this answer. – Adam Katz Jan 06 '17 at 02:54
Good answer as edited. Re: the dupe status, I'm quite sure that the underlying misconceptions behind the questions are the same, and that to answer one is to answer both -- but there probably is a case to be made that the particular instance marked as dupe doesn't pose that shared underlying question in the clearest manner. – Charles Duffy Jan 06 '17 at 03:17

score 0 · Answer 2 · answered Jan 05 '17 at 20:24

0

Use an array:

strings=('a a a' 'bcd' 'e e')
for i in "${strings[@]}"
do
 echo "$i"
done

answered Jan 05 '17 at 20:24

chepner

497,756
71
530
681

This answer uses [bash](https://en.wikipedia.org/wiki/Bash_(Unix_shell)). The question asks about [Bourne shell](https://en.wikipedia.org/wiki/Bourne_shell), which does not support list variables. – Adam Katz Jan 06 '17 at 01:30
1

The user isn't using Bourne shell, because Mac OS X does not come with Bourne shell. – chepner Jan 06 '17 at 01:53
I don't know what's installed on Christian's computer. I merely saw the lack of a bash tag or references to bash and I _did_ see a reference to "bourne shell." I don't know how hard it would be to install Almquist, Bsh, or any of the others on OS X; I assume it wouldn't be difficult (though I can't speak for whatever bashisms may exist in various OS components and apps). – Adam Katz Jan 06 '17 at 02:02
2

Most people who say they are using Bourne shell have no idea which shell they are using. – chepner Jan 06 '17 at 02:03
@AdamKatz, ...aside: Almquist *isn't* Bourne, it's POSIX sh. No version of Almquist, even back to the original 1989 USENET posting, had the characteristic Bourne behavior of treating `^` as a pipe character. Neither does bsh, for that matter. – Charles Duffy Jan 06 '17 at 02:29
...related: [Common Characteristics of Bourne Shells](http://www.in-ulm.de/~mascheck/bourne/common.html) – Charles Duffy Jan 06 '17 at 02:37
@CharlesDuffy – That list was intended as an example of very basic /bin/sh candidates (and it's nowhere near complete). Just don't bring up Solaris jsh (their /bin/sh, which lacks parameter expansion and `$(…)` command substitution). Bourne's use of `^` as `|` has always made me shudder. Thanks for the link. It in turn links to a GNU guide on [portable shell programming](https://www.gnu.org/savannah-checkouts/gnu/autoconf/manual/autoconf-2.69/html_node/Portable-Shell.html) which has some nice historical Bourne-vs-POSIX shell notes. – Adam Katz Jan 06 '17 at 19:16
@AdamKatz, ...hasn't had Solaris had a POSIX-compliant `/bin/sh` since 11? But yes, it was Bourne well into recent times. – Charles Duffy Jan 06 '17 at 23:45
@CharlesDuffy – I still remember when they shifted the major version to obsolete the "2" line (2.6 was succeeded by 7), but my Solaris knowledge stops around the introduction of ZFS and its intentional non-GPL-compatibility, so I can't speak to Solaris 11. I do recall working around the jsh/bsh shenanigans with a one-liner like `uname -s |grep SunOS >/dev/null && /usr/ucb/bin/sh "$0" "$@"` at the top of some of my scripts. I guess they need a `[ -x /usr/ucb/bin/sh ]` in that chain now. – Adam Katz Jan 07 '17 at 00:19
IIRC, the way autotools (and thus all GNU-toolchain-created `configure` scripts) looks for Bourne is checking whether `echo ^ cat` emits `^ cat` on its output. – Charles Duffy Jan 07 '17 at 01:30

score 0 · Answer 3 · answered Jan 06 '17 at 02:23

0

If you want shell-like string-splitting with quotes honored, use xargs:

str="'a a a' 'bcd' 'e e'"
echo "$str" | xargs printf '%s\n'

...will emit:

a a a
bcd
e e

However, in general, the Right Thing is to avoid program design that expects this behavior.

answered Jan 06 '17 at 02:23

Charles Duffy

280,126
43
390
441

bourne shell does unexpected word breaks

3 Answers3

Original answer