1

I want to read a file lines by line in Unix shell scripting. A line in the file can contain pretty much any kind and any number of characters. So far i've tried a simple read script

while read line
do
   echo $line
done < datafile

But if i had a trailing whitespaces this script was outputing part of the lines concatenated to eachother or even duplicated. So i've modified it to:

while IFS= read -r line; do
   echo $line
done < datafile

Which fixed the problem and after that, it was working fine. But when i encountered lines that contain special characters - french or german special characters, chinese, cyrilic etc - the script ended up again concatenating and/or duplicating them.

For example:

A file containing names of 4 PDFs(could be anything else), as it is visible in the console with cat command:

????????? ???.pdf 
AR_CLAIMS_BUBBLES.pdf
leur_compte__-_re??u_le.pdf
blomberg_RG62540.pdf

The output of the script for that file is:

 ????????? ???.pdf   AR_CLAIMS_BUBBLES.pdf   blomberg_RG62540.pdf
 AR_CLAIMS_BUBBLES.pdf
 leur_compte__-_re??u_le.pdf   blomberg_RG62540.pdf
 blomberg_RG62540.pdf leur_compte__-_re??u_le.pdf

I don't understand how or why this happen, but it seems to be highly dependant on these special characters. The script only malfunctions when handling lines with such characters(visible as '?' in console).

In that case, how can i accurately read the individual lines?

Note: unfortunately giving the actual content of the files is not possible as i have only access via console to the Unix system.

Smoothgoal
  • 25
  • 7
  • you probably need to quote the echo: `echo "$line"`. – fedorqui Aug 17 '16 at 11:14
  • Are you sure the file is using unix-style line endings (only \n)? You also should use a compatible locale. – lynxlynxlynx Aug 17 '16 at 11:15
  • Actually fedorqui is correct - it indeed is caused because of the missing quotes. But I really can't understand why. How is there such a huge difference just because of a simple quotes? – Smoothgoal Aug 17 '16 at 11:20
  • Check the related question I closed this as a duplicate to. It gives you a lot of details about this issue! – fedorqui Aug 17 '16 at 11:22

0 Answers0