0

I have several rows of data that looks the following:

255.255.255.255 -  - [dd/mm/yyyy hh:mm:ss] "GET /index.html?parameter1=valueN HTTP/1.1" 200 -
255.255.255.255 -  - [dd/mm/yyyy hh:mm:ss] "GET /index.html?parameter2=valueN HTTP/1.1" 200 -

However I want to only show this part of the line:

parameter=value

But it must be done by a command similar to this because I need the continual output that tails can deliver:

tails -f <file.log> | whatever

I can remove both the first part of the line and the last part, but I am unable to figure out how to achieve that in one line. I have the following script that will do as I want, but it only works with cat, and not with tails:

cat file.log | cut -d'?' -f2 | sed 's/ HTTP\/1\.1\" 200 \-/ /g' | grep -E "parameter1|parameter2"

Results in:

parameter1=valueN
parameter2=valueN

I have also tried several other commands with cut, tr and grep that brings me somewhat closer to what I want, however still doesn't match what I've shown above. What do I do?

tripleee
  • 175,061
  • 34
  • 275
  • 318
Frederik1111
  • 49
  • 1
  • 4
  • 2
    It's unclear what the problem is. I'm pretty sure you are simply a victim of buffering (this is a common FAQ) but I'm hesitant to close this as a duplicate until you can confirm that things work with the obvious fix. `tail -f file.log | sed -un 's%.* "GET /index\.html?\(parameter[12]=value[^ ]*\) HTTP/1\.[01]" 200 -.*%\1%p'` on GNU `sed`; e.g. MacOS requires `-l` instead of `-u` for unbuffered. – tripleee Aug 22 '19 at 12:40
  • Possible duplicate of [Redirecting tail output into a program](/questions/36525541/redirecting-tail-output-into-a-program) – tripleee Aug 22 '19 at 12:43
  • Maybe a better duplicate: https://stackoverflow.com/questions/7161821/how-to-grep-a-continuous-stream – tripleee Aug 22 '19 at 12:45
  • See: `man sed | grep buffer` and `man grep | grep buffer` – Cyrus Aug 22 '19 at 12:46
  • Thanks for your answers. I've also tried using commands to avoid the buffering problem, but yet tail doesn't allow me to pipe more than once! I've tried using "stdbuf -oL" before grepping as well as "--line-buffered" none of them solve my problem. – Frederik1111 Aug 22 '19 at 12:48
  • Upload file.log somewhere. – Cyrus Aug 22 '19 at 12:56
  • 1
    `sed` can contain multiple substitutions as separate steps, ie. `sed 's/1/X/;s/2/Y/;s/3/Z/;.....'` I don't see a need to all `grep -E` after using `sed`. Also, your example output implies you want to remove any numbers that are trailing after both `param` and `value`. ? That can be fixed with a `sed` command as well (if I understand your Q correctly). Good luck. – shellter Aug 22 '19 at 13:34
  • `echo "255.255.255.255 - - [dd/mm/yyyy hh:mm:ss] "GET /index.html?parameter1=valueN HTTP/1.1" 200 -" | sed 's/^.*[?]//;s/ .*$//;s/[0-9]//g'` produces `parameter=valueN`. Its left as a learning exercize for the reader to remove the `N` from `valueN` ;-) . Good luck. – shellter Aug 22 '19 at 13:45
  • 1
    I'm surprized that `stdbuf -oL` isn't working for you, (but my system doesn't have that, so I can't confirm the options you are using). I would run a second window that just displays the raw output from `tail -f` so you can see if there are pauses created by the producer program that you might not anticipate. AND update your Q with your best attempt including the call to `stdbuf`, as this is what you need to fix. AND your Q subject mentions `sort` but it's not in the body of your q (is it?) . A source file has to be closed for `sort` to work, if can't take input from `tail -f file`. GoodLuck. – shellter Aug 22 '19 at 13:48
  • Maybe you just have a problem with log rotation. Have you looked at `tail -F`? It will tell tail to not just wait for changes in the file but to also check if the file was moved and a new one with the same name was created (due to log rotation), and if so it will automatically reopen the new log file. – Robin479 Aug 22 '19 at 13:59

1 Answers1

0

Thank you @shellter!

The following command solved my problem:

tail -f file.log | sed 's/^.*[?]//;s/ .*$//;s/[0-9]//g'

However now I get "..." whenever there is an output that doesn't match the expression, I'd rather have nothing printed. Also there are issues with non-unicode characters that I will be looking more into. Thanks!

Frederik1111
  • 49
  • 1
  • 4