Concatenating string read from file with string literals creates jumbled output

Question

My problem is that the result is jumbled. Consider this script:

#!/bin/bash
INPUT="filelist.txt"

i=0;
while read label
do
    i=$[$i+1]
    echo "HELLO${label}WORLD"
done <<< $'1\n2\n3\n4'

i=0;
while read label
do
    i=$[$i+1]
    echo "HELLO${label}WORLD"
done < "$INPUT"

filelist.txt

5
8
15
67
...

The first loop, with the immediate input (through something I believe is called a herestring (the <<< operator) gives the expected output

HELLO1WORLD
HELLO2WORLD
HELLO3WORLD
HELLO4WORLD

The second loop, which reads from the file, gives the following jumbled output:

WORLD5
WORLD8
WORLD15
WORLD67

I've tried echo $label: This works as expected in both cases, but the concatenation fails in the second case as described. Further, the exact same code works on my Win 7, git-bash environment. This issue is on OSX 10.7 Lion.

How to concatenate strings in bash | Bash variables concatenation | concat string in a shell script

Answer 1

Well, just as I was about to hit post, the solution hit me. Sharing here so someone else can find it - it took me 3 hours to debug this (despite being on SO for almost all that time) so I see value in addressing this specific (common) use case.

The problem is that filelist.txt was created in Windows. This means it has CRLF line endings, while OSX (like other Unix-like environments) expects LF only line endings. (See more here: Difference between CR LF, LF and CR line break types?)

I used the answer here to convert the file before consumption. Using sed I managed to replace only the final line's carriage return, so I stuck to known guns and went for the perl approach. Final script is below:

 #!/bin/bash
INPUTFILE="filelist.txt"
INPUT=$(perl -pe 's/\r\n|\n|\r/\n/g' "$INPUTFILE")

i=0;
while read label
do
    i=$[$i+1]
    echo "HELLO${label}WORLD"
done <<< $'INPUT'

Question has been asked in a different form at Bash: Concatenating strings fails when read from certain files