58

I want to extract a substring where certain pattern exist from pipe separated file, thus I used below command,

awk -F ":" '/REWARD REQ. SERVER HEADERS/{print $1, $2, $3, $4}' sample_profile.txt

Here, 'REWARD REQ. SERVER HEADERS' is a pattern which is to be searched in the file, and print its first 4 parts on a colon separated line.

Now, I want to send bash variable to act as a pattern. thus I used below command, but it's not working.

awk -v pat="$pattern" -F ":" '/pat/{print $1, $2 , $3, $4 } sample_profile.txt

How can I use -v and -F in a single awk command?

fedorqui
  • 275,237
  • 103
  • 548
  • 598
Chintamani Manjare
  • 1,543
  • 1
  • 13
  • 28

3 Answers3

104

If you want to provide the pattern through a variable, you need to use ~ to match against it:

awk -v pat="$pattern" '$0 ~ pat'

In your case, the problem does not have to do with -F.

The problem is the usage of /pat/ when you want pat to be a variable. If you say /pat/, awk understands it as a literal "pat", so it will try to match those lines containing the string "pat".

All together, your code should be:

awk -v pat="$pattern" -F ":" '$0~pat{print $1, $2, $3, $4 }' file
#                             ^^^^^^

See an example:

Given this file:

$ cat file
hello
this is a var
hello bye

Let's look for lines containing "hello":

$ awk '/hello/' file
hello
hello bye

Let's now try looking for "pat", contained in a variable, the way you were doing it:

$ awk -v pat="hello" '/pat/' file
$                                    # NO MATCHES!

Let's now use the $0 ~ pat expression:

$ awk -v pat="hello" '$0~pat' file
hello                                 # WE MATCH!
hello bye

Of course, you can use such expressions to match just one field and say awk -v pat="$pattern" '$2 ~ pat' file and so on.

From GNU Awk User's Guide → 3.1 How to Use Regular Expressions:

When a regexp is enclosed in slashes, such as /foo/, we call it a regexp constant, much like 5.27 is a numeric constant and "foo" is a string constant.

And GNU Awk User's Guide → 3.6 Using Dynamic Regexps:

The righthand side of a ‘~’ or ‘!~’ operator need not be a regexp constant (i.e., a string of characters between slashes). It may be any expression. The expression is evaluated and converted to a string if necessary; the contents of the string are then used as the regexp. A regexp computed in this way is called a dynamic regexp or a computed regexp:

BEGIN { digits_regexp = "[[:digit:]]+" }
$0 ~ digits_regexp    { print }

This sets digits_regexp to a regexp that describes one or more digits, and tests whether the input record matches this regexp.

Jonathan Leffler
  • 730,956
  • 141
  • 904
  • 1,278
fedorqui
  • 275,237
  • 103
  • 548
  • 598
  • What about mixing a variable with text constants like "text${var}othertext" as pattern? – BlackEye Mar 03 '21 at 21:12
  • @BlackEye just use it normally: `awk -v pat="text${v}othertext" '$0 ~ pat' file`. – fedorqui Mar 04 '21 at 08:12
  • How can we use this when we later need access to matching sub-patterns? That is, in my `awk` script, I have `match($0, /^(Foo)(Bar)$/, m)` (that is, in case of match `m[1]` contains `Foo` and `m[2]` contains `Bar`), and I now would need something like `match($0, /^(Foo${Var})(Bar)$/, m)` (so that in case of match e.g. `m[1]` becomes `FooBoo` when `Var` is `Boo`). Does the `match` function only support constant patterns? – Binarus Sep 09 '22 at 13:54
  • @Binarus frankly, I don't know right now. I suggest asking a new question, since this may be of interest to many. – fedorqui Sep 12 '22 at 05:23
  • Actually, it's in the manual, but I was hoping for a more elegant and faster method. Since a variable's content can be embedded in a string, we can construct the pattern in string form and give that string to the `match()` function (instead of a real pattern). The manual states that this will cost performance (because the string must be scanned twice); furthermore, it is cumbersome (e.g. because ```\``` must be written as ```\\``` in the string). Apart from that, it works as expected. – Binarus Sep 13 '22 at 07:45
  • 1
    ...and in case anyone's wondering how to make it PART of a pattern, here's an example:`$0 ~ /^."$myvar"/ { }` – Sridhar Sarnobat Apr 17 '23 at 19:32
12
awk -v pat="$pattern" -F":" '$0 ~ pat { print $1, $2, $3, $4 }' sample_profile.txt

You can't use the variable inside the regex // notation (there's no way to distinguish it from searching for pat); you have to specify that the variable is a regex with the ~ (matching) operator.

Jonathan Leffler
  • 730,956
  • 141
  • 904
  • 1,278
  • The `{print $1, $2, $3, $4}` portion really helps me to realise that I can also print a specific column only. Thank you! :) – Glenn Mohammad May 10 '22 at 11:51
0

This is kind of a hack but it makes things a little simpler for me.

cmd="awk '/$pattern/'"
eval $cmd

making it a string first lets you manipulate it past the boundaries of awk

Ryan Donahoe
  • 17
  • 1