1

I don't quite know the term(s) for this part of the bash shell. In one specific, yet critical, case my script falls afoul of this effect:

Objective

Perform this command chain within a script where the awk expression (-e) comes form a variable. This example works when it is a script argument.

echo "test string" | awk  -e { print $0; }

Problem example

On the command line I am seeking to produce output of: "test string", viz.:

$ optE="-e "
$ argE="{ print \$0; }"
$ set -x; echo  "test string" | awk  $optE$argE ; set +x
+ awk -e '{' print '$0;' '}'
+ echo 'test string'
awk: cmd. line:1: {
awk: cmd. line:1:  ^ unexpected newline or end of string
+ set +x

In a way I can see what's happened. Is there a good/best way to not have the $argE variable tokenised after it is expanded?

Typing that same command on the command line works as you know:

$ echo "test string" | awk   -e '{ print $0; }'
test string

Because the expression is enclosed in single quote. I haven't found a way to make that happen using a variable...

$ optE="-e "
$ argE="'{ print \$0; }'"
$ set -x; echo  "test string" | awk  $optE$argE ; set +x
+ echo 'test string'
+ awk -e ''\''{' print '$0;' '}'\'''
awk: cmd. line:1: '{
awk: cmd. line:1: ^ invalid char ''' in expression
+ set +x

Needless to say, I'm on stackoverflow because the things I've tried and read in ohter questions, etc. Don't give desirable result.

  • Solutions welcome
will
  • 4,799
  • 8
  • 54
  • 90
  • does https://stackoverflow.com/a/44055875/9072753 answer your question? – KamilCuk Oct 08 '21 at 13:16
  • 1
    Could you stick your awk command into a file and then use `-f filename.awk`? Notice that your very first example is already producing an error because of lack of quotes around the awk command. – Benjamin W. Oct 08 '21 at 13:19
  • @BenjaminW. .. That's correct. In fact the `-f version` script path works fine (fingers crossed). This is a simple example from a script. Sometimes you just want type a short command, not edit a new file. – will Oct 08 '21 at 13:41

1 Answers1

2

Word splitting expansion is applied after the unquoted variable expansion (${var} or $var) is replaced by its expansion. Word splitting splits the result on (white-)spaces, no matter the quotes. No matter what you put inside the string, if the variable expansion is unquoted, the result will be word split. To affect word splitting, you have to change the way you expand the variable, not it's content (i.e. change $var to "$var").

Is there a good/best way to not have the $argE variable tokenised after it is expanded?

Yes, quote the expansion. Rule of a thumb: never $var always "$var". Also, check your scripts with shellcheck.

In your case, it's simple, just assign the variable the content you want to execute, and quote the expansions:

optE="-e"
argE='{ print $0; }'
echo "test string" | awk "$optE" "$argE" 
                         ^^^^^^^ - variable expansion inside quotes

For more cases, use bash arrays arr=(-e '{ print $0; }'), and properly awk "${arr[@]}" expand them.

Research: https://www.gnu.org/software/bash/manual/html_node/Shell-Operation.html , bash manual shell expansions , https://mywiki.wooledge.org/BashFAQ/050 https://mywiki.wooledge.org/Quotes https://mywiki.wooledge.org/BashPitfalls , When should I wrap quotes around a shell variable? https://github.com/koalaman/shellcheck/wiki/Sc2086 .

KamilCuk
  • 120,984
  • 8
  • 59
  • 111
  • I included the `set -x` debugging so people can see that the `$optE` part woks just fine. I also made the example as simple as I could so people could try it first. When you run this version you get several more single-quotes(`'`). There are a few questions on this same conundrum, it is feels a bit like a black art. – will Oct 08 '21 at 13:44
  • 1
    In this case, `$optE` should **not** be quoted. If the variable is empty, then the awk script body will be the empty string, and `"$argE"` will be taken as the data file name. Using shell arrays is definitely the right way to go. – glenn jackman Oct 08 '21 at 13:49
  • `When you run this version you get several more single-quotes(')` Run what version? `optE="-e" ; argE='{ print $0; }' ; echo "test string" | awk "$optE" "$argE"` outputs `test string` for me. – KamilCuk Oct 08 '21 at 13:51
  • It's not really a black art; the problem is that too many people assume that shell parameters behave like *variables*, when they are really more like *macros*. You don't pass variable values to commands; you let the shell expand the parameter (according to very specific rules), and the result of that expansion is the value (or values) that the command receives as arguments. – chepner Oct 08 '21 at 17:14
  • @chepner ... Thank you those are wise words and I can see how you are right. However in some cases we are making tools (or wishing to) to extend the environment. Then, the mission is not to work with the status quo, it is to get the designed/desired result. Or, find a satisfactory alternative. I'll take that advice on-board because understanding the rules, just makes the _extend_-ing cases a little better. – will Oct 15 '21 at 14:13