2

Here is how I read a file row by row:

while read ROW
do
...
done < file

I don't use the other syntax

cat file | while read ROW
do
...
done

because the pipe creates a subshell and makes me lose the environment variables. The problem arises if the file doesn't end with a newline: last line is not read. It is easy to solve this in the latter syntax, by echoing just a newline:

(cat file; echo) | while read ROW
do
...
done

How do I do the same in the former syntax, without opening a subshell nor creating a temporary file (the list is quite big)?

Narcolessico
  • 1,921
  • 3
  • 19
  • 19
  • I specify: with "losing env variables" I mean that the changes made inside the loop are not visible outside (e.g. a counter). Of course from within the loop I can still access the inherited variables. – Narcolessico Jan 18 '11 at 18:03
  • Maybe i am not simulating the result correctly, but what version of bash are you using? I am using bash4 and do have a problem reading the last line, with and without the last new line. – ghostdog74 Jan 19 '11 at 00:23
  • bash 4.1.5. Maybe you should check the kind of newlines (\r, \n or both, mac linux or win-style) – Narcolessico Jan 20 '11 at 11:22

4 Answers4

2
#!/bin/bash

while read ROW
 ...
done < <(cat file ; echo)
Ignacio Vazquez-Abrams
  • 776,304
  • 153
  • 1,341
  • 1,358
  • This does create a subshell, even though, it's the `cat` one rather than the `while` one (so it might be OK for the OP which probably didn't want the `while` to be in a subshell. (Also, it requires the `<()` piping which is not available in all shells.) – Oblomov Jan 18 '11 at 17:39
2

A way that works in all shells is the following:

#!/bin/sh

willexit=0
while [ $willexit == 0 ] ; do
read ROW || willexit=1
...
done < file

A direct while read will exit as soon as read encounters the EOF, so the last line will not be processed. By checking the return value outside the while, we can process the last line. An additional test for the emptiness of $ROW should be added after the read though, since otherwise a file whose last line ends with a newline will generate a spurious execution with an empty line, so make it

#!/bin/sh

willexit=0
while [ $willexit == 0 ] ; do
read ROW || willexit=1
if [ -n "$ROW"] ; then
  ...
fi
done < file
Oblomov
  • 761
  • 3
  • 10
  • I would change that `if` to `if [ -n "$ROW" ] || [ "$willexit" == 0 ]` so that empty lines could still be processed (note also the quotes around the variables). – Dennis Williamson Jan 18 '11 at 17:48
  • Good point about the quotes, but your `if` addition means "exit at the first empty line" instead of "skip empty lines". if you want to process empty lines, just get rid of the if (first version of the code) – Oblomov Jan 18 '11 at 17:58
0

The POSIX way to do this is via a named pipe. 

#!/bin/sh

[ -p mypipe ] || mkfifo mypipe

(cat num7; echo) > mypipe & 

while read line; do 
  echo "-->$line<--"
  export CNT=$((cnt+1))
done < mypipe

rm mypipe

echo "CNT is '$cnt'"

Input

$ cat infile
1
2
3
4
5$

Output

$ (cat infile;echo) > mypipe & while read line; do echo "-->$line<--"; export CNT=$((cnt+1)); done < mypipe; echo "CNT is '$cnt'"
[1] 22260
-->1<--
-->2<--
-->3<--
-->4<--
-->5<--
CNT is '5'
[1]+  Done                    ( cat num7; echo ) > mypipe
SiegeX
  • 135,741
  • 24
  • 144
  • 154
  • Doesn't this still cause env vars set in the while loop to not be seen after it, which I assume was the reason why the OP didn't want subshells? – Oblomov Jan 18 '11 at 17:59
  • @Oblomov After I posted it I realized my answer didn't make sense since env vars from parent->subshell should always work. I realize he is talking about the other way around. Answer updated to make this work with a POSIX solution. – SiegeX Jan 18 '11 at 18:08
0

From an answer to a similar question:

while IFS= read -r LINE || [ -n "${LINE}" ]; do
   ...
done <file

The IFS= part prevents read from stripping leading and trailing whitespace (see this answer).

If you need to react differently depending on whether the file has a trailing newline or not (e.g., warn the user) you'll have to make some changes to the while condition.

Community
  • 1
  • 1
Richard Hansen
  • 51,690
  • 20
  • 90
  • 97