script not reading last line of a file

Question

i have a file created in windows using notepad:

    26453215432460
    23543265235421

    38654365876325


    12354152435243

I have a script which will read every line, and create a command like below in other file for every line and will not consider blank lines:

    CRE:EQU,264532154324600,432460,1;

Now if I save my input file after hitting enter after the last line of number 12354152435243, then the output file consists the command above corresponding to all numbers(including the last 12354152435243:

    CRE:EQU,264532154324600,432460,1;
    CRE:EQU,235432652354210,235421,1;
    CRE:EQU,386543658763250,876325,1;
    CRE:EQU,123541524352430,435243,1;

but if I save the file, without hitting enter after the last number is keyed in i.e after this 12354152435243, then after the script executes, I don't see the output file have the command for the last number:

    CRE:EQU,264532154324600,432460,1;
    CRE:EQU,235432652354210,235421,1;
    CRE:EQU,386543658763250,876325,1;

Can somebody explain the error in the code:

    while read LINE
    do
    [ -z "$LINE" ] && continue
    IMEI=`echo $LINE | sed 's/ //g' | sed -e 's/[^ -~]//g'`
    END_SERIAL=`echo $IMEI | cut -c9- | sed 's/ //g' | sed -e 's/[^ -~]//g'`
    echo "CRE:EQU,${IMEI}0,${END_SERIAL},${list},,${TODAY};" >> /apps/ins/list.out
    done < "${FILE_NAME}"

kindly help

Answer 1

Use

grep . "${FILE_NAME}" | while read LINE

or

while read LINE
do
....
done < <(grep . "${FILE_NAME}")

The grep is less sensible to line-ending, and you will get empty-line skip for a free... :)

Honestly, never tried windows, all above is OK for unix...

EDIT Explanation:

make the next file:

echo -n -e 'line\n\nanother\nno line ending here>' >file.txt

the file contains 4 lines (although the last "line" is not a "correct" one)

line

another
no line ending here>

Usual shell routines, as read or wc looking for line ending. Therefore,

$ wc -l file.txt 
         3 file.txt

When you grepping for '' (empty string) the grep returns every line where found the string, so

$ grep '' file.txt

prints

line

another
no line ending here>

When grep prints out the found lines - ensures than one `\n' exists at the end, so

$ grep '' file.txt | wc -l

returns

4

therefore, for these situations, is better to use grep with -c (count) and not wc.

$ grep -c '' file.txt
4

Now, the . dot. The dot mean any character. So, when you grepping for a ., you get all lines what contain at least one character. And therefore, it will skip all lines what doesn't contain any character = skips empty lines. So,

$ grep . file.txt
line
another
no line ending here>

again, with added line ending to the last line (and skipped the empty line). Remember, the (space) is character too, so when the line contains only one space it is NOT EMPTY. Counting non-empty lines

$ grep . file.txt | wc -l
      3

or faster

$ grep -c . file.txt
3

Answer 2

If you do a help read it says for -d delim continue until the first character of DELIM is read, rather than newline.

So read will continue until it hits a \n or if you specify -d delim.

So you probably need to change the delim or you can try read -e

Answer 3

read will read untill a new line has been found and when it finds a new line, it will return you the line. But if the file ends without a new line, read treats this as an error. So, even if read has set the returning variable with the line read till now, the return code of read is set to indicate an error. Now the while read ... this loop body will only executes if the command executes with a success which is not a case here. Thus you miss the last line.

For overcoming this , you can change the condition to also check the returning variable is empty or not. Hence the condition succeeds even if read fails, as the variable is already set till the end of the file.

This is not related to line ending in different OS, i mean it's somehow related, but the exact root cause is always the read fails to find a new line at the end of the line/file, and the last line is missing the loop body.

Below is an example

[[bash_prompt$]]$ echo -ne 'hello\nthere' > log
[[bash_prompt$]]$ while read line; do  echo $line; done < log
hello
[[bash_prompt$]]$ while read line || [ -n "$line" ]; do  echo $line; done < log
hello
there
[[bash_prompt$]]$

Answer 4

read nees the end of line to read the input. Try

echo -n $'a\nb' | while read x ; do echo $x ; done

It only prints a.

Answer 5

To prevent script not reading last line of a file:

cat "somefile" | { cat ; echo ; } | while read line; do echo $line; done

Source : My open source project https://sourceforge.net/projects/command-output-to-html-table/