2

I have this for:

for i in `ls -1 access.log*`; do tail $i |awk {'print $4'} |cut -d: -f 1 |grep - $i > $i.output; done

ls will give access.log, access.log.1, access.log.2 etc.
tail will give me the last line of each file, which looks like: 192.168.1.23 - - [08/Oct/2010:14:05:04 +0300] etc. etc. etc
awk+cut will extract the date (08/Oct/2010 - but different in each access.log), which will allow me to grep for it and redirect the output to a separate file.

But I cannot seem to pass the output of awk+cut to grep.

The reason for all this is that those access logs include lines with more than one date (06/Oct, 07/Oct, 08/Oct) and I just need the lines with the most recent date.

How can I achieve this?

Thank you.

w00t
  • 616
  • 3
  • 9
  • 16
  • Can you post some example input and expected output? – Fred Foo Oct 26 '10 at 11:29
  • 1
    One problem is that you keep overwriting the same output for each access.log. Note that you can also use multiple files for tail. – Tomas Oct 26 '10 at 11:31
  • lol.. you're right, don't know how I overlooked this fact. Let me think about it so I can come up with a workaround. – w00t Oct 26 '10 at 11:35
  • just add a counter which is appended to the log name – joni Oct 26 '10 at 11:39
  • @tripleee thanks for the feedback, but this question was asked in 2010. – w00t Sep 21 '16 at 14:15
  • We mark as duplicate in order to collect the relevant information in one place. The clarity of the question and the answers is more important than their age. – tripleee Sep 21 '16 at 14:27

5 Answers5

1

why don't you break it up into steps??

for file in *access.log
do
  what=$(tail "$i" |awk {'print $4'} |cut -d: -f 1)
  grep "$what" "$file" >> output
done
ghostdog74
  • 327,991
  • 56
  • 259
  • 343
  • This, too, overwrites `output` in every iteration. – Fred Foo Oct 26 '10 at 11:37
  • I guess I can break it up, but wanted to run it from CLI. Anyhow, It's been more than one time that I needed to use a feature like this, so I'm trying to find the answer for my scenario. – w00t Oct 26 '10 at 11:55
  • The elegant solution is to redirect the `done` to overwrite the file; this way, the entire loop body is redirected, and the file is opened for writing just once, intead of repeatedly inside the loop. – tripleee Sep 21 '16 at 14:29
1

As a sidenote, tail displays the last 10 lines.

A possible solution would be to grepthis way:

for i in `ls -lf access.log*`; do grep $(tail $i |awk {'print $4'} |cut -d: -f 1| sed 's/\[/\\[/') $i > $i.output; done
mouviciel
  • 66,855
  • 13
  • 106
  • 140
  • I don't quite care about this aspect. I could have used tail -n 1. – w00t Oct 26 '10 at 12:22
  • @mouviciel - I already tried it exactly as you wrote it, but it gives grep: Unmatched [ or [^ – w00t Oct 26 '10 at 12:44
  • I see, `[` has a special meaning for `grep`. I edit my answer. – mouviciel Oct 26 '10 at 12:51
  • sed 's#\\[##g' - this one works. Nonetheless, I'm still curious about my initial question – w00t Oct 26 '10 at 13:06
  • It seems that this is the only solution, to write all the other commands inside the one you need. So, in my case, it is: for i in \`ls -lf access.log*`; do grep $(tail $i |awk {'print $4'} |cut -d: -f 1| sed 's#\[##g') $i > $i.output; done. Thanks for the support. – w00t Oct 26 '10 at 13:31
  • The output from `ls -l` is a big long line; you do not want to pass that whole thing to `tail`! – tchrist Oct 31 '10 at 17:31
  • why not? you think that tail is gonna get sick? fine, ls -1 instead of -l. this wasn't actually the problem at hand. – w00t Nov 02 '10 at 08:15
1

You shouldn't use ls that way. Also, ls -l gives you information you don't need. The -f option to grep will allow you to pipe the pattern to grep. Always quote variables that contain filenames.

for i in access.log*; do awk 'END {sub(":.*","",$4); print substr($4,2)}' "$i" | grep -f - $i > "$i.output"; done

I also eliminated tail and cut since AWK can do their jobs.

Dennis Williamson
  • 346,391
  • 90
  • 374
  • 439
  • I see no problem with ls -l, it's how I've got used to use it. I could have used ls -1, but it is so not important. AWK isn't a good candidate for tail and cut's job because AWK reads the whole file to get to the END, and when one has 500+mb of files, it takes too long. Now about the grep -f, that really is helpful. Thanks. – w00t Nov 02 '10 at 08:19
  • 1
    http://mywiki.wooledge.org/ParsingLs explains why `ls` is problematic. – tripleee Sep 20 '16 at 05:27
0

you can try:

 grep "$(stuff to get piped over to be grep-ed)" file

I haven't tried this, but my answer applied here would look like this:

 grep "$(for i in `ls -1 access.log*`; do tail $i |awk {'print $4'} |cut -d: -f 1 |grep - $i > $i.output; done)" $i
gideonite
  • 1,211
  • 1
  • 8
  • 13
0

Umm... Use xargs or backticks.

man xargs

or http://tldp.org/LDP/Bash-Beginners-Guide/html/sect_03_04.html , section 3.4.5. Command substitution

joni
  • 5,402
  • 1
  • 27
  • 40
  • I don't see how xargs can help me. It would need to know what the output from all the prior commands is and then pass it as a pattern to grep. It would not be `command | xargs grep 123 file.txt`, but it would be `command | xargs grep -pattern-from-command- file.txt` – w00t Oct 26 '10 at 12:40