18

Writing a shell script that pretty much wraps around an Awk script. I'd like to have my syntax colouring on in the awk section so I feel there must be a better way to embed awk scripts to bash than just quote the script in.

Or is it the best way to develop the awk on a separate file and use awk -f in the bash script until the script's done? I'd like to keep within one file all times!

#!/bin/bash
awkscript='
BEGIN{
    print "test"
}
{print $3}'
df | awk "$awkscript"

This is the quote way, and quite ugly if you ask me. Any chance for heredocs here?

mike3996
  • 17,047
  • 9
  • 64
  • 80

6 Answers6

29

Yes, in Bash this is possible in a slightly less ugly way:

#!/bin/bash
df | awk -f <(cat - <<-'EOD'
        BEGIN{
            print "test"
        }
        {print $3}
EOD
)

The <() code block is replaced with a path (something like /dev/fd/63) which points to a file descriptor tied to the output of the code. Make sure the awk script is indented with tabs and not spaces (<<- strips leading tabs).

Another way to do this is to hide the awk script at the end of the bash script like this (don't forget to exit):

#!/bin/bash
df | awk -f <(sed -e '0,/^#!.*awk/d' $0)
exit $PIPESTATUS

#!/usr/bin/awk -f
BEGIN {
    print "test"
}
{print $3}
mss
  • 1,804
  • 1
  • 17
  • 18
  • 1
    This is a cool solution. With different `sed` commands, you could embed multiple sub-scripts. Excellent! – mike3996 Oct 25 '11 at 17:00
  • 2
    I prefer the cat variant for multiple scripts because the awk code stays near the command executed. I use the latter for shell scripts which are more or less wrappers around awk/sed scripts. See eg. https://github.com/mss/sed-header-filter/blob/master/header-filter.sh for an extended (`sed`) example I once whipped up. – mss Oct 28 '11 at 11:58
  • why is the df required? cant figure that out. – alvin Apr 05 '12 at 09:33
  • 2
    The `df` is not required but the example command which is awk'ed here. Replace `df` with any command you like, same for the awk script of course. – mss Apr 10 '12 at 13:52
11

If you don't want to use stdin for the data, do this --- it's portable, fast, and doesn't use any bashisms.

awk -f- inputfile.txt <<'EOF'
BEGIN {
  print "hello world"
}
EOF

Telling awk to use an input filename of - causes it to read the script from stdin.

Of course, you now have to store your data in a file. If you really can't do this, use a named pipe.

f=/tmp/$$.data
trap "rm -f $f" EXIT
mknod $f p
(df > $f) &
awk -f- $f <<EOF
{ print }
EOF
David Given
  • 13,277
  • 9
  • 76
  • 123
  • Good answer. But you must use single quotes or your field variables won't parse correctly: `awk -f- inputfile.txt <<'EOF'`. – Steve May 23 '16 at 02:16
  • Yup --- actually figured this out for myself about ten minutes after posting, then forgot to update my answer! Thanks. – David Given May 23 '16 at 15:18
6

ugly too:

read -r -d '' awkscript <<'EOF'
BEGIN{ print "test"}
{print $3}
EOF
df | awk "$awkscript"
clt60
  • 62,119
  • 17
  • 107
  • 194
  • it's still way better that to close the whole script in single quotes. :) – mike3996 Jun 27 '11 at 09:29
  • @Fredrik's answer contained the compulsory manual entry for heredocs. They are naturally redirected to something's stdin. AWK already uses the stdin for reading the actual data, so there's no better way than this. – mike3996 Jun 27 '11 at 12:55
  • But oopsy woopsy -- this solution also makes bash substitute for `$3` so you need to escape the dollar. Not that aesthetic after all! And 9 rep short of fixing the minor issue :) – mike3996 Jun 27 '11 at 13:00
  • 1
    Hm.. strange, because the <<'EOF' (EOF in the apostrophes) mean *don't do variable expansion*. Are your sure than you got expanded the "$3" as a shell variable? Or ensure than you have EOF in the apostrophes. – clt60 Jun 27 '11 at 13:26
  • My noobish bad. Copypasted to experiment and somewhere in the middle I somehow managed to remove the quotes. Sorry :F – mike3996 Jun 27 '11 at 13:36
3

Sometimes this may be usefull:

awkscript=$(cat << EOT
BEGIN{
    print "test"
}
{print $3}
EOT
)

df | awk "$awkscript"
Sylvain Leroux
  • 50,096
  • 7
  • 103
  • 125
3

Not sure what you mean, but are you looking for something like this?

#!/bin/bash

awk 'BEGIN {
    print "hello world"
}
{ 
    print $3
}' <<< "$(df)"

Have a look at 3.6.7 Here Strings in the bash manual

Fredrik Pihl
  • 44,604
  • 7
  • 83
  • 130
  • Not what I had in mind, but +1'd because the manual led me to the right direction: not possible to have multiple stdin reads. – mike3996 Jun 27 '11 at 12:54
0

One way is to let awk read rules from stdin using (heredoc or here string) but process real files or files created by subprocess substitution, like this

awk -f- <(df) <<'EOF'
BEGIN{
    print "test"
}
{print $3}
EOF

-f- should appear before <(df). The delimiter for the heredoc should be quoted so that the $ signs will be passed into awk instead of being expanded.

doraemon
  • 2,296
  • 1
  • 17
  • 36