20

I usually use the read command to read an input file to the shell script line by line. An example code such as the one below yields a wrong result if a new line isn't inserted at the end of the last line in the input file, blah.txt.

#!/bin/sh

while read line
do
echo $line
done <blah.txt

So if the input file reads something like -

One 
Two
Three
Four

and I do not hit return after Four, the script fails to read the last line, and prints 

One
Two
Three

Now if I leave an extra blank line after Four, like,

One 
Two
Three
Four
//blank line

the output prints all the lines, including Four. However, this is not the case when I read a line using the cat command; all lines including the last get printed without me having to add an extra blank line at the end.

Anyone has ideas on why this happens? The scripts I create will mostly be run by others, so it isn't necessary they're going to add an extra blank line at the end of every input file.

I've been trying to figure this out for ages; I'd appreciate it if you have any solutions(of course, the cat command is one, but I'd like to know the reason behind read not working as well).

Jahid
  • 21,542
  • 10
  • 90
  • 108
Caife
  • 415
  • 1
  • 5
  • 16

5 Answers5

21

read reads until it finds a newline character or the end of file, and returns a non-zero exit code if it encounters an end-of-file. So it's quite possible for it to both read a line and return a non-zero exit code.

Consequently, the following code is not safe if the input might not be terminated by a newline:

while read LINE; do
  # do something with LINE
done

because the body of the while won't be executed on the last line.

Technically speaking, a file not terminated with a newline is not a text file, and text tools may fail in odd ways on such a file. However, I'm always reluctant to fall back on that explanation.

One way to solve the problem is to test if what was read is non-empty (-n):

while read -r LINE || [[ -n $LINE ]]; do
  # do something with LINE
done

Other solutions include using mapfile to read the file into an array, piping the file through some utility which is guaranteed to terminate the last line properly (grep ., for example, if you don't want to deal with blank lines), or doing the iterative processing with a tool like awk (which is usually my preference).

Note that -r is almost certainly needed in the read builtin; it causes read to not reinterpret \-sequences in the input.

rici
  • 234,347
  • 28
  • 237
  • 341
  • I like the `grep .` solution. Nice and simple. – paddy Jun 24 '13 at 04:51
  • @rici what does `||` and `[[ -n $LINE ]]` mean here? does `||` mean Or? – gkmohit Jul 16 '14 at 14:31
  • @Unknown: It means "or" in the sense that `cmd1 || cmd2` succeeds if either `cmd1` succeeds or `cmd2` succeeds. (Success means an exit code of 0.) `read` succeeds if it encounters a newline character; `[[` succeeds if the test expression (in this case `$LINE` is not empty) is true. – rici Jul 16 '14 at 21:46
  • Well done, you may want to mention what the `-n` test actually tests for -- it tests if the length of the string is non-zero. Same as `[[ ! -z "${LINE}" ]]`. –  Mar 14 '16 at 21:28
8
DONE=false
until $DONE
do
  read line || DONE=true
  echo $line
done < blah.txt

How to use `while read` (Bash) to read the last line in a file if there’s no newline at the end of the file?

Community
  • 1
  • 1
Zombo
  • 1
  • 62
  • 391
  • 407
8

Use while loop like this:

while IFS= read -r line || [ -n "$line" ]; do
  echo "$line"
done <file

Or using grep with while loop:

while IFS= read -r line; do
  echo "$line"
done < <(grep "" file)

Using grep . instead of grep "" will skip the empty lines.

Note:

  1. Using IFS= keeps any line indentation intact.

  2. You should almost always use the -r option with read.

  3. Don't read lines with for

  4. File without a newline at the end isn't a standard unix text file.

Jahid
  • 21,542
  • 10
  • 90
  • 108
4

One line answer:

IFS=$'\n'; for line in $(cat file.txt); do echo "$line" ; done
sandino
  • 3,813
  • 1
  • 19
  • 24
  • won't work if you have spaces in the line - will treat as a separate line – schmi Jan 18 '15 at 16:33
  • Thanks schmi for your comment you are right, I edited the answer and putted IFS=$'\n'; first this tell cat command to only use newline as separator. The answer remains one line :) – sandino Jan 19 '15 at 13:30
  • This changes the `IFS` of the current shell. [And you shouldn't read lines with for](http://mywiki.wooledge.org/DontReadLinesWithFor) – Jahid Jul 14 '15 at 05:01
  • You can change IFS later with unset IFS, about efficiency for performs good it only requires more memory like twice the file size because it hold it entirely en memory but with today's advances memory is cheap and not a problem. I agree with you that with big files is better to use a while, my answer was just to show a one line approach to the problem that is something that many people want: simplicity – sandino Jul 15 '15 at 13:15
0

Below code with Redirected "while-read" loop works fine for me

while read LINE
do
      let count++
      echo "$count $LINE"

done < $FILENAME

echo -e "\nTotal $count Lines read"
Aniket Thakur
  • 66,731
  • 38
  • 279
  • 289