Use output of sed for a while read loop

Question

I want the while loop to ignore blank (empty) lines and lines that contain #

I tried including sed -e 's/#.*$//' -e '/^$/d' on the input file and piping this into the while loop. This did not work.

file=$(sed -e 's/#.*$//' -e '/^$/d' foo.txt)

while IFS=: read -r f1 f2 f3 f4; do 

Command

done <"$file"

score 4 · Accepted Answer · edited Mar 28 '19 at 16:23

You're trying to open that output as a filename, which it almost certainly isn't.

Run sed in a subshell and redirect its output to while loop instead.

while IFS=: read -r f1 f2 f3 f4; do
    # do something
done < <(sed -e 's/#.*$//' -e '/^$/d' foo.txt)

For anyone interested in how the syntax used in this answer works, see the bash-hackers' wiki on process substitution, and for why this syntax is better than sed ... | while read ..., see BashFAQ #24.

score 0 · Answer 2 · answered Mar 28 '19 at 16:18

0

You can use grep:

grep -Ev '^\s*$|^\s*#' foo.txt | while IFS=: read -r f1 f2 f3 f4; do 
    Command
done

answered Mar 28 '19 at 16:18

accdias

Even `grep -E` isn't guaranteed to support `\s`; only the regex syntax documented in the POSIX ERE standard is available across all platforms, and `\s` -- like other backslash-sequence shorthand -- isn't there. To have guaranteed support on all POSIX platforms, use `[[:space:]]` instead. – Charles Duffy Mar 28 '19 at 16:21
...so, the use of `grep` doesn't fix anything (the old `sed` command worked fine), and (with the use of `\s`) introduces portability issues; the use of a pipe *does* fix the OP's problem, but it fixes it in a way that introduces [different bugs](http://mywiki.wooledge.org/BashFAQ/024). – Charles Duffy Mar 28 '19 at 16:22

score -2 · Answer 3 · answered Mar 28 '19 at 16:10

-2

use grep -v:

-v, --invert-match        select non-matching lines

answered Mar 28 '19 at 16:10

Dimas

That certainly works, but the problem isn't with the `sed` code, it's with how they're getting its result into the `while read` loop. Thus, this answer replaces one piece of working code with a *different* piece of working code, but doesn't actually fix the bug the OP is asking about. – Charles Duffy Mar 28 '19 at 16:13
If you read the title, then yes. But this is the first line of the post: "I want the while loop to ignore blank (empty) lines and lines that contain #" OP tried with sed, I'm suggesting to use grep instead. – Dimas Apr 04 '19 at 17:08
Here's the thing, though -- *the thing the OP tried with sed works fine*. You're suggesting that they change the part that already works, not the part that's broken. The bug isn't caused by `file=$(sed -e 's/#.*$//' -e '/^$/d' foo.txt)` doing something other than what the OP expects it to, so changing that line to `file=$(grep -v '#|^$')` or whatever doesn't fix it. – Charles Duffy Apr 04 '19 at 17:12
...instead, the problem is caused by `<"$file"` iterating over a *file named by the output of* their `sed` command, or your `grep` command; switching which of those two interchangable commands is used is irrelevant to whether they hit that issue unless the other, later code is fixed. – Charles Duffy Apr 04 '19 at 17:15

3 Answers3