1

I'm a grep and sed newbie, and have read through a bunch of answers on SO referring to grepping IPs in apache logs with no luck for my particular situation.

I have megs of error logs from bots and nefarious humans hitting a site, and I need to search through the logs and find the most common IPs so I can confirm they're bad and block them in .htaccess.

But, my error logs don't have the IP as the first item on the line as it seems most Apache logs do, according to the other answers here on SO. In my logs, the IP is within each line and in the format [client 123.456.78.90].

This older answer is exactly what I need, I think, Grepping logs for IP adresses as it "will print each IP... sorted prefixed with the count."

But according to the answerer, "It assumes the IP-address is the first thing on each line."

How can I modify the sed command from that answer for the IP format [client 123.456.78.90] rather than the IP on the first line of each log entry?

sed -e 's/\([0-9]\+\.[0-9]\+\.[0-9]\+\.[0-9]\+\).*$/\1/' -e t -e d access.log | sort | uniq -c

8/25/14 This works re: Kent's answer below:

grep -o '[0-9]\+\.[0-9]\+\.[0-9]\+\.[0-9]\+' logfile|sort|uniq -c

Update 9/02/14

To sort by number of occurrences of each IP;

grep -o '[0-9]\+\.[0-9]\+\.[0-9]\+\.[0-9]\+' logfile|sort -n | uniq -c | sort -rn
Community
  • 1
  • 1
markratledge
  • 17,322
  • 12
  • 60
  • 106
  • have you tried that line? the sed line doesn't need the ip in the beginning of line. there is no `^` in regex. what it gives you if you add `.*` after `'s/`? like `'s/.*\([0-9].....` – Kent Aug 24 '14 at 20:28
  • Yes, I've tried it; I get nothing returned in OS X bash. Is there error reporting in sed? – markratledge Aug 24 '14 at 20:32
  • is there only one ip in each line of log? If true, can you pls try this? `grep -o '[0-9]\+\.[0-9]\+\.[0-9]\+\.[0-9]\+' logfile|sort|uniq -c` – Kent Aug 24 '14 at 20:36
  • Ah, nice! That works. I don't know enough to know the difference between sed and grep, though. Add it as an answer below. – markratledge Aug 24 '14 at 20:40

2 Answers2

3

grep is for Globally finding Regular Expressions on individual lines and Printing them (G/RE/P get it?).

sed is for Stream EDiting (SED get it?), i.e. making simple substitutions on individual lines.

For any other general text manipulation (including anything that spans multiple lines) you should use awk (named after 3 guys who ran out of imagination for naming tools).

awk '
    match($0,/[0-9]+\.[0-9]+\.[0-9]+\.[0-9]+/) { cnt[substr($0,RSTART,RLENGTH)]++ }
    END { for (ip in cnt) print cnt[ip], ip }
' logfile
Ed Morton
  • 188,023
  • 17
  • 78
  • 185
  • 1
    I was going to make the next Facebook using `sed` and you're saying I shouldn't do that. Scratch!! Back to the drawing board again!! +1 for pointing me to right direction. – jaypal singh Aug 24 '14 at 23:25
  • 1
    Thanks! That works, though slightly differently in that it doesn't sort by IP or number of occurances of the IP. I can see that awk's syntax may be clearer for me to learn. – markratledge Aug 25 '14 at 04:14
  • With GNU awk you can control the output order by setting the `PROCINFO["sorted_in"]` variable. – Ed Morton Aug 25 '14 at 06:44
2

dirty and quick :

grep -o '[0-9]\+\.[0-9]\+\.[0-9]\+\.[0-9]\+' logfile|sort|uniq -c

a big diff between sed and grep is: sed can change the input text (like substitution), but grep can't. :-)

Kent
  • 189,393
  • 32
  • 233
  • 301
  • I was able to learn how to add "sort" (I added this to my question) to sort by IP occurrence. But now, (regarding your notes regarding the difference between sed and grep), due to server configs I can't change, I now need to exclude lines that contain "zlib" in the sorted output. Is excluding lines something for grep; or sed or awk? – markratledge Sep 04 '14 at 16:07