Extract two substrings from line in bash then concatenate them

Question

This solution will be part of a script.

Given an error log, I need to extract the failed status/reason for failure and the username, then concatenate them so the output can be sent in an email.

My output should look something like this:

Operation status: failed,Job Description not updated because this is not a matching Job Description ID.,username=FOO
Operation status: failed,Job Description not updated because this is not a matching Job Description ID.,username=BAR

Each line in the error log file closely resembles this:

"{ Operation status: failed,Job Description not updated because this is not a matching Job Description ID.,Sent Data:{lastAppraisalScore=0.0, country=null, jobTitle=LABORER..., username=FOO},...sendAcctActNotif=N}}"

There are no ellipses in the file; each line contains considerably more than is shown, but I have shown only the important parts.

Can you share your research efforts on this and did you try debugging the script with the `set -x` option? — Inian, Apr 18 '16 at 14:13
Is each line valid JSON? If so, you could probably achieve this by passing each line to `jq` and selecting the `Operation status` and `username` keys. — Tom Fenech, Apr 18 '16 at 14:16
I have not tried debugging with the `set -x` option. `egrep -o "Operation status:" input` partially returns what I need, but need to extract until the first `.,` so that I may pick up the entire error code. — sharpmartin6, Apr 18 '16 at 14:19

tripleee · Answer 1 · 2016-04-18T17:34:15.957

2

A simple sed substitution would seem more adequate.

sed -n '/^"{ Operation status: failed,/!b
    s///;s/,Sent.*, username=/\t/;s/}.*//p' file

This searches for the first expression; if it is not found, we bypass this line. Otherwise, we substitute the matched string with nothing, and proceed to substitute away the other strings we don't want to keep, and finally print what remains.

edited Apr 18 '16 at 17:34

answered Apr 18 '16 at 16:05

tripleee

175,061
34
275
318

I get this error when I try to run the code: `sed: can't find label for jump to \`s///'`. – sharpmartin6 Apr 18 '16 at 16:58
Try adding a semicolon after the `b`. I would have expected all `sed` dialects to acoept a newline there as statement separator, but apparently not. – tripleee Apr 18 '16 at 17:33
However, I added a missing `!` (not) – tripleee Apr 18 '16 at 17:34

score 1 · Accepted Answer · edited May 23 '17 at 11:59

Using this answer you can extract some part of your line (if it always looks the same way). Then you can process your file line by line :

while read line; do
  status=`grep -oP '(?<=Operation status:).*?(?=Sent)' <<< "$line"`
  user=`grep -oP '(?<=username=).*?(?=})' <<< "$line" | head -n 1`
  echo "Operation status:" $status "username="$user
done < file.txt

My example file.txt :

{ Operation status: failed1,Job Description not updated because this is not a matching Job Description ID.,Sent Data:{lastAppraisalScore=0.0, country=null, jobTitle=LABORER..., username=FOO1}, {username=BAR1}...sendAcctActNotif=N}}
{ Operation status: failed2,Job Description not updated because this is not a matching Job Description ID.,Sent Data:{lastAppraisalScore=0.0, country=null, jobTitle=LABORER..., username=FOO2}, {username=BAR2}...sendAcctActNotif=N}}
{ Operation status: failed3,Job Description not updated because this is not a matching Job Description ID.,Sent Data:{lastAppraisalScore=0.0, country=null, jobTitle=LABORER..., username=FOO3}, {username=BAR3}...sendAcctActNotif=N}}

My output :

Operation status: failed1,Job Description not updated because this is not a matching Job Description ID., username=FOO1
Operation status: failed2,Job Description not updated because this is not a matching Job Description ID., username=FOO2
Operation status: failed3,Job Description not updated because this is not a matching Job Description ID., username=FOO3

This works, thank you much. But, I just realized that each line of the error log has two instances of `username=`, so both are being returned! How can I return only the first instance of the username in each line? — sharpmartin6, Apr 18 '16 at 15:27
@sharpmartin6 you can use | head -n 1 to keep the first element (first line). Look at my edit — Fabich, Apr 18 '16 at 15:47
Thank you for the quick response. `| head -n 1` returns only the first element in the file. Is it possible to return only the first instance of username in each line with `grep`? — sharpmartin6, Apr 18 '16 at 15:52

Extract two substrings from line in bash then concatenate them

2 Answers2