Skip rows based on date check from a STDOUT in shell

Question

I am running a shell script under which i am running program which is writing output data on STDOUT and from there i am capturing the output and redirecting it into a file. But that program writes a lot of data which i need to filter out when i am writing it into a file

test -d $data | grep "^[A-Z]*|[0-9]*" >> $OUT_FILE

Now in this output on row 3 i am getting a value on which i want to have a check if that values is less than 43564 then skip the row.

test -d $data | grep "^[A-Z]*|[0-9]*" | awk -F "|" 'if($3 >= 43564){print $0}'>> $OUT_FILE

But this is not working for me , suggest me what wrong i am doing

`test -d "$data"` doesn't write anything to standard output, and so there is nothing for `grep` to read. — chepner, Oct 19 '20 at 13:14

RavinderSingh13 · Accepted Answer · 2020-10-19T13:17:55.903

Could you please try following, since there are no samples given so couldn't test it. As an important change you need to change from => to >= in your awk condition. Then I have also fixed your if condition it ideally an if condition should be inside {if(your condition)....} braces.

Actually speaking we need not to have if condition we could simply run conditions itself and no need to mention print function too, because awk works on method of regexp/condition then action so if this condition is TRUE it will print that line(since no action mentioned and default action of awk is printing the current line).

As per @chepner comment comment test -d is NOT writing anything to standard output so changed it to a condition now.

if [[ test -d $data ]]
then
     awk -F "|" '/^[A-Z]*|[0-9]*/ && ($3 >= 43564)' Input_file > $OUT_FILE
else
     echo "Please check $data directory NOT present."
fi

NOTE: We need to use grep with awk, since awk is capable of checking regex since this is OP's effort and I am going for OP's attempt fix here.

Thanks for the `=>` fix; shamelessly copied that into my answer. +1 — tripleee, Oct 19 '20 at 13:22
I'm hoping so. I missed it myself but figured our answers were different enough that mine was worth keeping. — tripleee, Oct 19 '20 at 17:53
After looking at the answer i realized that my approach was not correct . — Developer, Oct 19 '20 at 18:37

tripleee · Answer 2 · 2020-10-19T13:25:53.290

You can't grep the output from test because it doesn't print anything.

Probably you want

test -d "$data" ||
awk -F "|" '/^[A-Z]*|[0-9]*/ && $3 >= 43564' "$data" >> "$OUT_FILE"

which is equivalent to the more beginner-friendly

if ! [ -d "$data" ]; then
   awk -F "|" '/^[A-Z]*|[0-9]*/ && $3 >= 43564' "$data" >> "$OUT_FILE"
fi

You almost never want to pipe grep to Awk; this is a very minor antipattern, but also very easy to avoid. See useless use of grep. Somewhat similarly, {print $0} is already the default action in Awk, so we can simply leave that out, and just specify conditions for when to print.

If you don't want to examine the file $data in the then part, obviously replace that part with the name of the file you want to filter. (Notice also proper quoting.) If you are filtering standard input, you can eminently well take out the file name and pipe to the entire if expression.

some complex process which generates this output |
if ! [ -d "$data" ]; then
    awk -F "|" '/^[A-Z]*|[0-9]*/ && $3 >= 43564'
fi >>"$OUT_FILE"

Skip rows based on date check from a STDOUT in shell

2 Answers2