4

Hi I have a shell command like this.

s3=$(awk 'BEGIN{ print "S3 bucket path" }
 /Executing command\(queryId/{ sub(/.*queryId=[^[:space:]]+: /,""); q=$0 }
 /s3:\/\//{ print "," $10 }' OFS=',' hive-server2.log)

The output of the above command like this.

echo $s3

2018-02-21T17:58:22,
2018-02-21T17:58:26,
2018-02-21T18:05:33,
2018-02-21T18:05:34

I want to select the last line only. I need the last output like this.

 2018-02-21T18:05:34

I tried like this.

awk -v $s3 '{print $(NF)}'

Not working.Any help will be appreciated.

Teju Priya
  • 595
  • 3
  • 8
  • 18

5 Answers5

7

In general, command | tail -n 1 prints the last line of the output from command. However, where command is of the form awk '... { ... print something }' you can refactor to awk '... { ... result = something } END { print result }' to avoid spawning a separate process just to discard the other output. (Conversely, you can replace awk '/condition/ { print something }' | head -n 1 with awk '/condition/ { print something; exit }'.)

If you already have the result in a shell variable s3 and want to print just the last line, a parameter expansion echo "${s3##*$'\n'}" does that. The C-style string $'\n' to represent a newline is a Bash extension, and the parameter expansion operator ## to remove the longest matching prefix isn't entirely portable either, so you should make sure the shebang line says #!/bin/bash, not #!/bin/sh

Notice also that $s3 without quotes is an error unless you specifically require the shell to perform whitespace tokenization and wildcard expansion on the value. You should basically always use double quotes around variables except in a couple of very specific scenarios.

Your Awk command would not work for two reasons; firstly, as explained in the previous paragraph, you are setting s3 to the first token of the variable, and the second is your Awk script (probably a syntax error). In more detail, you are basically running

awk -v s3=firstvalue secondvalue thirdvalue '{ print $(NF) }'
          ^ value    ^ script to run   ^ names of files ...

where you probably wanted to say

awk -v s3=$'firstvalue\nsecondvalue\nthirdvalue' '{ print $(NF) }'

But even with quoting, your script would set v to something but then tell Awk to (ignore the variable and) process standard input, which on the command line leaves it reading from your terminal. A fixed script might look like

awk 'END { print }' <<<"$s3"

which passes the variable as standard input to Awk, which prints the last line. The <<<value "here string" syntax is also a Bash extension, and not portable to POSIX sh.

tripleee
  • 175,061
  • 34
  • 275
  • 318
  • Grabbed the left content with left_stuff=`echo $s3 | sed "s/${s3##**$'\n'}//"` – archcutbank Aug 27 '21 at 13:26
  • @archcutbank That looks wrong for a couple of reasons. Without the quotes, there is no way the output from `echo` can contain a newline. (As already mentioned in the third paragraph, that's a common beginner error.) The double `**` is just redundant. I'm pretty sure you can accomplish that with just a parameter expansion anyway. Not sure what you mean by "left stuff"; what you have will remove what's on the left. Are you looking for `"${s3%%$'\n'*}"` to pick out the first line? Or `"${s3%$'\n'*}"` to get everything except the last? – tripleee Aug 27 '21 at 14:28
  • It just used sed to take off the last line and left you everything else (left of the last line). – archcutbank Aug 29 '21 at 20:18
  • The backticks disappeared because that's how you mark up `code` in comments. I mean you need `echo "$s3"` with double quotes around the string variable. And as noted in my previous comment, you don't need `sed` for this, any more than for the opposite operation. In shell scripting, you generally want to avoid calling external processes when you can. – tripleee Aug 30 '21 at 05:03
3

much simple way is

command | grep "your filter" | tail -n 1

or directly

command | tail -n 1
aze2201
  • 453
  • 5
  • 12
1

You could try this:

echo -e "This is the first line \nThis is the second line" | awk 'END{print}'
MagnusO_O
  • 1,202
  • 4
  • 13
  • 19
Anirudh Bharadwaj
  • 21
  • 2
0

another approach can be, processing the file from the end and exiting after first match.

tac file | awk '/match/{print; exit}'
karakfa
  • 66,216
  • 7
  • 41
  • 56
-1

Hi you can do it just by adding echo $s3 | sed '$!d'

s3=$(awk 'BEGIN{ print "S3 bucket path" }/Executing command\(queryId/{ sub(/.*queryId=[^[:space:]]+: /,""); q=$0 } /s3:\/\//{ print "," $10 }' OFS=',' hive-server2.log)

echo $s3 | sed '$!d'

It will simply print:-

2018-02-21T18:05:34

Hope this will help you.

Abhijit Pritam Dutta
  • 5,521
  • 2
  • 11
  • 17
  • Without quotes, the `echo` will only ever produce a single line. See [When to wrap quotes around a shell variable?](https://stackoverflow.com/questions/10067266/when-to-wrap-quotes-around-a-shell-variable) – tripleee Aug 27 '21 at 14:15