Repeated patterns in the log file

Question

Content of the log file is as below,

I, [2018-03-07T10:21:26.315447 #7531]  INFO -- : POST http://127.0.0.1:8080/api/v3/internal/allowed 6.69490
I, [2018-03-07T10:21:26.315799 #7531]  INFO -- : gitlab-shell: executing git command <git-upload-pack /var/opt/gitlab/git-data/repositories/GitLab-write-logs/write-logs.git> for user with key key-1.
I, [2018-03-07T10:23:00.567841 #7739]  INFO -- : POST http://127.0.0.1:8080/api/v3/internal/allowed 5.55714
I, [2018-03-07T10:23:00.568227 #7739]  INFO -- : gitlab-shell: executing git command <git-upload-pack /var/opt/gitlab/git-data/repositories/GitLab-read-logs/read-logs.git> for user with key key-3.
I, [2018-03-07T10:23:49.932948 #7851]  INFO -- : gitlab-shell: executing git command <git-receive-pack /var/opt/gitlab/git-data/repositories/GitLab-RW-logs/RW-logs.git> for user with key key-5.
I, [2018-03-07T10:55:29.533385 #4778]  INFO -- : gitlab-shell: executing git command <git-receive-pack /var/opt/gitlab/git-data/repositories/GitLab-write-logs/write-logs.git> for user with key key-1.
I, [2018-03-07T10:55:29.810478 #4790]  INFO -- : POST http://127.0.0.1:8080/api/v3/internal/allowed 0.10971

Pattern = < git-upload-

There are muliple lines with this matching pattern. How can I extract information from each line.

For a example, in the second line we have matching pattern < git-upload- then I would like to extract string '/var/opt/gitlab/git-data/repositories/GitLab-write-logs/write-logs from that line and assign to a variable and do some action.

I would like this process to be done to all matched lines sequentially without repeating the same lines, How should I do that? I appreciate everyone's input

Suppose, we have 50 lines with matching patterns lines in them. Can we grep one line one time do action and pick up next line and repeat the same.

logic:

    if more somefile.log | grep -Eq < git-upload-pack ; then
    variable = /var/opt/gitlab/git-data/repositories/GitLab-write-logs/write-logs [ .git shouldn't be there] 
       do some action
    fi

should I put while loop (i=0, i++)to repeat the process on each lines like ..check the line if pattern there then execute "if" condition and exit the condition and repeat for next line I am new to scripting and struggling to connect the dots

Thank you all

Ian McGowan · Answer 1 · 2018-04-10T21:02:32.703

Sounds like a job for grep and cut, as long as the format is predictable:

$grep git-upload-pack t | cut -d " " -f 13 | cut -d ">" -f 1
/var/opt/gitlab/git-data/repositories/GitLab-write-logs/write-logs.git
/var/opt/gitlab/git-data/repositories/GitLab-read-logs/read-logs.git

The grep git-upload-pack command finds lines that contain git-upload-pack
cut -d " " -f 13 grabs the 13th space-delimited field
cut -d ">" -f 1 grabs the 1st of 2 fields, effectively removing the trailing ">"

To loop over the resulting output:

$grep git-upload-pack t |
 cut -d " " -f 13       |
 cut -d ">" -f 1        |
 while read f; do echo your-command ${f%.*}; done

Where you can replace "your-command" with whatever you want to do with each match, and remove the echo to actually run it. Try it out with the echo until it makes sense.

How can remove the extension of a filename in a shell script? explains the ${f%.*} to remove .git.

Hi, think we hav 50 lines with matching patterns lines in them. Can we grep each line one time do action and pick up next line and repeat the same if more somefile.log | grep -Eq ` — Prs, Apr 06 '18 at 20:59
A while loop is a good idea. It would be even better if you showed something you'd tried to do already. There are a lot of bash questions about "read variable loop" - try googling for them. In the meantime, updated the answer with a hint. — Ian McGowan, Apr 10 '18 at 20:52

score -1 · Answer 2 · answered Apr 06 '18 at 21:16

-1

With gnu sed

sed '/<git-upload-/!d;s/.*<git-upload-[^/]*\([^>]*\).*/echo \1 | tr [:lower:] [:upper:] /e' infile

answered Apr 06 '18 at 21:16

ctac_

2,413
2
7
17

Repeated patterns in the log file

2 Answers2