Getting rid of some special symbol while reading from a file

Question

I am writing a small script which is getting some configuration options from a settings file with a certain format (option=value or option=value1 value2 ...).

settings-file:

SomeOption=asdf
IFS=HDMI1 HDMI2 VGA1 DP1
SomeOtherOption=ghjk

Script:

for VALUE in $(cat settings | grep IFS | sed 's/.*=\(.*\)/\1/'); do
    echo "$VALUE"x
done

Now I get the following output:

HDMI1x
HDMI2x
VGA1x
xP1

Expected output:

HDMI1x
HDMI2x
VGA1x
DP1x

I obviously can't use the data like this since the last read entry is mangled up somehow. What is going on and how do I stop this from happening?

Regards

Yet another CRLF issue :D Anyway... run `dos2unix settings` & you are good to go. If `dos2unix` is not available, use `sed -i 's/\r$//' settings` — anishsane, Jun 17 '16 at 07:28
That's DOS line endings. Run `dos2unix` command on the file. If you do not have "dos2unix", or run `tr -d '^M' < input_file` - make sure you enter ^M by keystrokes `Ctrl+v+m` — ring bearer, Jun 17 '16 at 07:31
The colleagues are using Windows and they want to edit the file as well, so I guess I'll go with the additional `sed` command as it's the easiest and most obvious method while keeping the file as it is. — cronotk, Jun 17 '16 at 07:35
`tr -cd "[:print:]\n" < file` should remove all non-printable characters from the file. Also btw your script demonstrates perfect UUOC http://stackoverflow.com/questions/11710552/useless-use-of-cat :) — Inian, Jun 17 '16 at 07:38

hek2mgl · Answer 1 · 2016-06-17T07:39:46.177

Generally you can use awk like this:

awk -F'[= ]' '$1=="IFS"{for(i=2;i<=NF;i++)print $i"x"}' settings

-F'[= ] splits the line by = or space. The following awk program checks if the first field, the variable name equals IFS and then iterates trough column 2 to the end and prints them.

However, in comments you said that the file is using Windows line endings. In this case you need to pre-process the file before using awk. You can use tr to remove the carriage return symbols:

tr -d '\r' settings | awk -F'[= ]' '$1=="IFS"{for(i=2;i<=NF;i++)print $i"x"}'

Kusalananda · Answer 2 · 2016-06-17T07:54:35.503

The reason is likely that your settings file uses DOS line endings.

Once you've fixed that (with dos2unix for example), your loop can also be modified to the following, removing two utility invocations:

for value in $( sed -n -e 's/^IFS.*=\(.*\)/\1/p' settings ); do
  echo "$value"x
done

Or you can do it all in one go, removing the need to modify the settings file at all:

tr -d '\r' <settings |
for value in $( sed -n -e 's/^IFS.*=\(.*\)/\1/p' ); do
  echo "$value"x
done

Getting rid of some special symbol while reading from a file

2 Answers2