Bash while loop stripping first part of command until last iteration of a line in file

Question

I am tying to do something quite simple: Run a command for every line in a file.

So far I have this (and have also tried different permutations of this):

URL=$1
COMMENT=${URL##*/}

curl $URL | while read DOMAIN || [[ -n $DOMAIN ]];
    do
        echo $DOMAIN
        echo "pihole -w $DOMAIN --comment $COMMENT -nr"

        pihole -w $DOMAIN --comment $COMMENT -nr
    done

The echos are there for debugging. What is happening is the following:

It goes line by line through a file hosted on GitHub and for each of those lines it runs: pihole -w $DOMAIN --comment $COMMENT -nr - every line is a domain.

However it only runs the full command with the pihole -w $DOMAIN portion on the last line, the preceding lines only fire with --comment $COMMENT -nr. As you can see from the output here:

t.ly
 --comment url_shorteners.txt -nr
 is not a valid argument or domain name!
adf.ly
 --comment url_shorteners.txt -nr
 is not a valid argument or domain name!
www.adf.ly
 --comment url_shorteners.txt -nr
 is not a valid argument or domain name!
bit.ly
 --comment url_shorteners.txt -nr
 is not a valid argument or domain name!
www.bit.ly
 --comment url_shorteners.txt -nr
 is not a valid argument or domain name!
goo.gl
 --comment url_shorteners.txt -nr
 is not a valid argument or domain name!
ow.ly
 --comment url_shorteners.txt -nr
 is not a valid argument or domain name!
www.ow.ly
 --comment url_shorteners.txt -nr
 is not a valid argument or domain name!
amzn.to
 --comment url_shorteners.txt -nr
 is not a valid argument or domain name!
amzn.com
pihole -w amzn.com --comment url_shorteners.txt -nr
  [i] amzn.com already exists in whitelist, no need to add!

As you can see from the echo that prints the command about to be run, for some reason the first part is stripped.

Am I missing something in terms of escaping or piping?

Thank you in advance!

It looks like the domain list you're downloading has DOS/Windows-style line endings, which can cause all sorts of confusion (see ["Are shell scripts sensitive to encoding and line endings?"](https://stackoverflow.com/questions/39527571/are-shell-scripts-sensitive-to-encoding-and-line-endings)). Try putting `set -x` at the beginning to get an execution trace, and you'll see a `\r` at the end of the `$DOMAIN` value. In this case, I'd use `... while IFS=$'\r' read ...` to trim it. BTW, use [shellcheck.net](https://www.shellcheck.net) to spot other potential problems in your script. — Gordon Davisson, Mar 10 '22 at 03:01
@GordonDavisson this was highly informative and thank you for the link as well, have saved that. Could you please add this as a main answer so that I can mark it as such? — alphadmon, Mar 12 '22 at 21:29

score 0 · Accepted Answer · answered Mar 14 '22 at 06:55

For future answer seekers.

Gordon Davisson had the answer, I was editing the file between my windows machine and my Linux machine. Windows appended \r to the end of lines in the file.

Adding the following to my loop fixed the issue:

while IFS=$'\r' read

A handy website and tip that Gordon Davisson provided was the following site: Shell Check.

Bash while loop stripping first part of command until last iteration of a line in file

1 Answers1