Bad/illegal format or missing URL - Passing variable value inside custom curl

Question

I am using a Linux shell (Ubuntu 20.04) in my Windows 10 Machine. I have a bash script that is supposed to

Go through a .txt file that contains a list of IPs
Curl each IP using a site https://ipinfo.io/ for a modified curl output

#!/bin/bash  
input="/mnt/d/Docs/BR Resources/iplist.txt"  
while IFS= read -r line  
do   
 curl https://ipinfo.io/$line  
done < "$input"

The expected output for each IP in the loop is :

{ "ip": "xx.xx.xx.xx",
"hostname": "c0241.brsol.com",
"city": "Ashburn",
"region": "Virginia",
"country": "US",
"loc": "xx.xxxx,xx.xxxx", "org": "BR, Inc.",
"postal": "xxxxx",
"timezone": "America/New_York",
"readme": "https://ipinfo.io/missingauth" }

However, what I get as the output is this error :

curl: (3) URL using bad/illegal format or missing URL

Example excerpt of iplist.txt

36.13.6.167
252.125.137.71
204.50.68.40
136.122.209.112
203.47.25.30
223.96.93.56
64.137.82.169
11.183.223.40
199.169.87.25
119.198.39.119

The issue seems to be that the value of $line is not being passed alongside the rest of the curl command.

How can I fix this to get the modified Curl output for each IP in this .txt file?

Run this loop with echo first: `while IFS= read -r line; do echo "https://ipinfo.io/$line"; done < "$input"` — Ivan, May 19 '21 at 07:50
@Ivan Hi, that gives the whole command as the output. Like this : "https://ipinfo.io/36.13.6.167" ( https:// should come in front of this but StackOverFlow seems to ignore that part when it's before an IP address) — Sandun, May 19 '21 at 07:57
@Sandun Just tested your code and seems to work as-is. Bash version 5.0.17(1)-release. — Bayou, May 19 '21 at 07:58
Is either the script or iplist.txt file in DOS/Windows format (see [this question](https://stackoverflow.com/questions/39527571/are-shell-scripts-sensitive-to-encoding-and-line-endings))? If that's not the problem, try putting `set -x` at the beginning of the script will make it print what's happening as it runs, which can help figure out what's going wrong. — Gordon Davisson, May 19 '21 at 08:00
@Bayou I have the exact same Bash version. However I am running this code in the Ubuntu Shell in my Win 10 machine. — Sandun, May 19 '21 at 08:00
Change the shebang to `#! /bin/bash -x` to see more details about the commands being executed. It will most likely be a windows problem :-) — Bayou, May 19 '21 at 08:01
Try to wrap data into array to remove all nonprintable junk then use arayvar in curl, like this: `while IFS= read -r line; do ip=($line); curl "https://ipinfo.io/$ip"; done < "$input"` — Ivan, May 19 '21 at 08:08
@Bayou I added the -x to the shebang and this is the line I got for the curl: curl $'https://ipinfo.io/36.13.6.167\r/' — Sandun, May 19 '21 at 08:49
@Sandun That `\r` indicates there's a (normally invisible) carriage return character at the end of the URL, probably because the ipinfo.txt file is in DOS/Windows format. If you have the `dos2unix` command, it'll convert the file to unix format. You could also use `while IFS=$'\r' read ...` to get `read` to strip the carriage return for you. The [question I linked earlier](https://stackoverflow.com/questions/39527571/are-shell-scripts-sensitive-to-encoding-and-line-endings) has lots more details and options. — Gordon Davisson, May 19 '21 at 09:03
@Sandun or like this: `while read line; do ip=($line); curl "https://ipinfo.io/$ip"; done < "$input"` — Ivan, May 19 '21 at 09:21
@GordonDavisson Thank you! Your last suggestion did the trick. Stripping the \r made it work. Can you post it as an answer so that I can accept it? — Sandun, May 19 '21 at 09:29
@Sandun It's probably better to have all the options for dealing for DOS/Windows files in one place, so I added it as an answer under that other question and I'll mark this as a duplicate so everyone that finds your Q will be directed there. — Gordon Davisson, May 19 '21 at 22:46

score 1 · Answer 1 · answered May 19 '21 at 08:18

I did below steps on my windows machine which generated the output:
Converted the file to dos2unix.exe ./test.sh

test.sh:

# !/usr/bin/bash
input="iplist.txt"  
while IFS= read -r line  
do   
 curl https://ipinfo.io/$line  
done < "$input"

Ouput recieved:

% Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100   300  100   300    0     0    300      0  0:00:01 --:--:--  0:00:01   532
{
  "ip": "36.13.6.167",
  "hostname": "kd036013006167.ppp-bb.dion.ne.jp",
  "city": "Saitama",
  "region": "Saitama",
  "country": "JP",
  "loc": "35.9081,139.6566",
  "org": "AS2516 KDDI CORPORATION",
  "postal": "330-0853",
  "timezone": "Asia/Tokyo",
  "readme": "https://ipinfo.io/missingauth"
}  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current

bash version : GNU bash, version 4.3.42(4)-release

Hi, can you elaborate on the second line? Converted the file to dos2unix.exe ./test.sh ? Does that mean I have to install something called dos2unix in my Ubuntu Shell (inside the Windows machine) ? — Sandun, May 19 '21 at 08:51
@Sundun I saw that your shell script was in the DOS format (\r / Carriage return CRLF) and not unix format.May be you used an editor like notepad++ which made the file in dos format, so i converted it to a unix file by the operation dos2unix. This file then would work properly on unix .Dos format is: CRLF (each line ends with CR then LF), Unix format is : LF only (each line ends with an LF char). You can install this in ubuntu using : `sudo apt-get install dos2unix ` — Altaf, May 19 '21 at 14:32

Bad/illegal format or missing URL - Passing variable value inside custom curl

1 Answers1