Extracting data from only one line in a command substitution without echo $data |

Question

I have a command that I'd like to run and get output from. Here's the command:

sh execute_odi_script.sh Audit_ExecutionLogStart.sql heelo

It should return

Begin Informatica Script 18 Redshift Delta copy completed at: 04/10/17 18:46:20 END

Now I need to grab just the output from the SQL file, which is 18 in this case.

Someone was able to help me achieve that, but only if I do it in 2 steps like so:

> logStart=$( sh execute_odi_script.sh Audit_ExecutionLogStart.sql heelo ) 
> echo $logStart | sed -r s/.*Begin\ Informatica\ Script\ \(\.*\)\ Redshift\ Delta.*/\\1/ ); echo $logStart
18

I've been trying to play around with how I'm doing the command substitution but am having trouble understanding what to even change. How do I do all this in one line?

As an aside, `echo $logStart` is innately buggy -- it expands wildcards in your data; string-splits on any kind of whitespace and rejoins on spaces (so it changes newlines and tabs to regular spaces); and otherwise munges your data. See [BashPitfalls #14](http://mywiki.wooledge.org/BashPitfalls#echo_.24foo), and consider making a habit of running your code through http://shellcheck.net/. — Charles Duffy, Apr 10 '17 at 18:56
`logStart=$(sh execute_odi_script.sh Audit_ExecutionLogStart.sql heelo | sed -r s/.*Begin\ Informatica\ Script\ $\.*$\ Redshift\ Delta.*/\\1/ ))` should do the trick (ignore the trailing a ending asterix, I can. The pipe (|) redirects the output of one command to the input of another one. The backticks (``) work for storing the output of a command into a variable. (Edited taking into account @CharlesDuffy's remark - thanks) — Viliam Aboši, Apr 10 '17 at 18:58
@ViliamAboši, backticks are present for backwards compatibility -- the preferred modern syntax (and yes, it's POSIX-defined, not a bashism) is `$()`. It's much easier to nest -- `foo $(bar $(baz) )` is much less trouble than its backtick-based equivalent. — Charles Duffy, Apr 10 '17 at 18:58
@ViliamAboši, ...in particular, using literal backslashes inside backticks is problematic (and, I note, your code above is attempting to do so), since they need to be doubled. See http://stackoverflow.com/questions/9405478/command-substitution-backticks-or-dollar-sign-paren-enclosed — Charles Duffy, Apr 10 '17 at 18:59
@CharlesDuffy I was doing the double backticks because of the markup. — Viliam Aboši, Apr 10 '17 at 19:01
Waitaminute. How does this differ from your other question at http://stackoverflow.com/questions/43329771/is-there-a-correct-way-to-parse-output-in-bash-from-a-command ? — Charles Duffy, Apr 10 '17 at 19:13

Charles Duffy · Answer 1 · 2017-04-10T19:18:00.207

0

First -- if you only care about one line, grep for that line:

logStart=$( sh execute_odi_script.sh Audit_ExecutionLogStart.sql heelo \
            | grep -e "Begin Informatica Script" \
            | sed -r 's/.*Begin Informatica Script (.*) Redshift Delta.*/\1/' )
echo "$logStart"

Alternately, you the entire pipeline inside the command substitution:

logStart=$( sh execute_odi_script.sh Audit_ExecutionLogStart.sql heelo \
            | tr '\n' ' ' \
            | sed -r 's/.*Begin Informatica Script (.*) Redshift Delta.*/\1/' )
echo "$logStart"

If you want to avoid the echo as a separate step, you might add a | tee /dev/stderr to the end of the pipeline (again, inside $()).

edited Apr 10 '17 at 19:18

answered Apr 10 '17 at 18:55

Charles Duffy

280,126
43
390
441

I've tried putting the entire pipeline inside the command substitution, but still don't get the proper output. If I do `logStart=$( sh execute_odi_script.sh Audit_ExecutionLogStart.sql heelo | sed -r 's/.*Begin Informatica Script (.*) Redshift Delta.*/\1/' ); echo $logStart` then it outputs the entire line still, and if I do `logStart=$( sh execute_odi_script.sh Audit_ExecutionLogStart.sql heelo | sed -r 's/.*Begin Informatica Script (.*) Redshift Delta.*/\1/' ); echo "$logStart" ` then it'll print 4 lines, `19` being one of them – simplycoding Apr 10 '17 at 19:06
Heh. So you are actually **depending on** the behavior of `echo $logStart` combining all your lines into one. Ugh. – Charles Duffy Apr 10 '17 at 19:08
Well, use `tr '\n' ' '` for that. Or, y'know, don't use `sed` for jobs it's not good at. (`awk`, for instance, will ignore lines that don't match a pattern by default). – Charles Duffy Apr 10 '17 at 19:08
Well, how do I check the actual value of `logStart` without `echo`? – simplycoding Apr 10 '17 at 19:12
Why do you think I'm telling you not to use `echo`? – Charles Duffy Apr 10 '17 at 19:14
@simplycoding that means that the original output of the execute_odi_script.sh has 4 lines and the number you want to get is the second line? If yes, you could just do `sh execute_odi_script.sh Audit_ExecutionLogStart.sql heelo | head -n 2 | tail -n 1` – Viliam Aboši Apr 10 '17 at 19:14

Extracting data from only one line in a command substitution without echo $data |

1 Answers1