For loop reading word instead of lines in bash

Question

I'm having a problem for an apparent simple script.

Basically it looks like this :

for s in $(cat FILENAME| grep User)
do
echo "$s"

done

But instead of giving me result something like :

User bla bla bla
User2 something blabla

I'm having as a result :

User
bla
bla
bla
User2

If you read the bash manual, `for x in ...` loops over *words*, not lines. It's the wrong tool for this job. — Gordon Davisson, Jun 13 '19 at 18:53
@GordonDavisson: not necessarily. Spaces are often a problem, also if you iterate over files or find results you might encouter spaces. It would be bad, if you couldn't use "for" in all those cases. And that's where IFS comes in handy (see below). — jottbe, Jun 13 '19 at 19:48
You're asking for the correct way to write a shell loop to read the output of grep and then output exactly what grep would output without the loop. Don't do that for the obvious reason that it's pointless and you probably shouldn't be using this approach at all for whatever it is you're really trying to do. You should read https://unix.stackexchange.com/q/169716/133219 and several FAQs if you're considering doing this. Also google UUOC! — Ed Morton, Jun 13 '19 at 22:25

jottbe · Accepted Answer · 2019-06-15T19:35:05.727

As @donjuedo statet, space is usually regarded as a separator, that's why you don't get the whole lines. There are several ways to work around this. I list the solutions for both reading from file and from the output of a command. As input you can create a file with the following content and name it testfile.txt (with empty lines, lines with spaces in between and also on both ends:

This is the first line
2
third line

fifth line 
 sixth line

Solution 1: most generally applicable

while IFS= read -u 3 -r line; do
    echo ">${line}<"
    read -p "press enter to show next line" var
    echo "read caught the following input: >$var<"
done 3< testfile.txt

The variant that reads from a pipe looks exactly the same, only the last line changes. As an example I process all lines from the testfile that contain spaces, which eliminates lines two and four (here with <&3 instead of -u 3 - both are equivalent):

while IFS= read -r line <&3;  do
   ...
done 3< <(grep " " testfile.txt)

This works for large input files. The 3< indirection seems awkward, but makes sure, that processes within the loop still can read from standard input (see the read statement "press enter..."). This might be important if you execute commands that might show user prompts themselfes (like rm etc). So:

works for large input files as well without blowing up memory
stdin is not redirected (instead fd 3 is used)
spaces inbetween, and at both ends of lies are retained (using IFS)
empty lines are retained es well

Thanks to @BenjaminW for proposing this.

Solution 2: with IFS and for (with restrictions)

OFS="$IFS"
IFS=$'\n'
for line in $(cat testfile.txt) ; do
  echo ">${line}<"
  read -p "press enter to show next line" var
  echo "read caught the following input: >$var<"
done
IFS="$OFS"

This temporarily changes the field separator to newline, so the file is only splitted on line breaks.

not recommended for large input because of the command line substitution ($(cat testfile))
stdin is also not redirected, so it is possible to use stdin in the body without restricitons
spaces inbetween, and at both ends of lies are retained (using IFS)
empty lines as line four are skipped here (line legth 0 / matching ^$)
if you use this, you have to make sure, that you reset IFS
it might get messy, if your loop body needs a different interpretation of fields (e.g. needs to read something else which should be split around spaces).

Thanks it worked : cat apm | grep User| while read ligne do echo "$ligne" done — khaled83, Jun 13 '19 at 19:52
This is subject to pathname expansion; you could suppress that with `set -f`. It also skips blank lines; see [Why you don't read lines with "for"](https://mywiki.wooledge.org/DontReadLinesWithFor). So, all in all, it's still best practice to use something like `while IFS= read -r line; do ... done < <(grep ... infile)` instead. — Benjamin W., Jun 13 '19 at 22:11
See also https://stackoverflow.com/questions/11710552/useless-use-of-cat — tripleee, Jun 14 '19 at 08:09
@BenjaminW.: thanks for the link, I'll change my description. But if I compare the code in your comment with the code linked by another user, I recognize that you use stdin. So in your solution user interaction (also by automatic queries from rm or other commands) would not be possible and interfere with the loop logic. — jottbe, Jun 14 '19 at 08:29
I'd do that with `read -r -u3 line; do ...; read var; ...; done 3< <(grep ... infile)`. — Benjamin W., Jun 14 '19 at 13:46
@BenjaminW.: thank you, I changed it. This seems to be the best solution. — jottbe, Jun 15 '19 at 08:21
You have to add the `-u3` on the `while` line to make `while` read from that file descriptor. — Benjamin W., Jun 15 '19 at 18:20

score 0 · Answer 2 · answered Jun 13 '19 at 18:29

0

The cat and grep are producing lines, which happen to contain spaces. The for construct is making $s from each piece, not each line.

From what you've shown, you could simply use the cat and grep, and not bother with the loop.

answered Jun 13 '19 at 18:29

donjuedo

2,475
18
28

For loop reading word instead of lines in bash

2 Answers2