0

I'm having a weird problem with running a simple curl command inside a while loop. I have a list of links in a txt file which I want to process. The problem is that I am getting the bad/illegal URL error:

curl: (3) URL using bad/illegal format or missing URL

Here is the code (Accessions.txt has the list of URLs, one per line):

awk -F "," '{print $12}' "$INPUT" | sed 's,/[^/]*$,,' | sort | uniq > Accessions.txt
while read -r line; do curl -I "$line"; done < Accessions.txt

All links have this pattern: https://foo.bar.org/xxxx/xxxxxxxxxxx

What's weird is that if I pass the links one by one in curl they work fine e.g. curl -I https://foo.bar.org/xxxx/xxxxxxxxxxx

Any suggestions?

AlexNikolaus
  • 3
  • 2
  • Are there no invisible characters like spaces or carriage returns? Also try using a different `fd` for the `read`. – Jetchisel Jul 11 '21 at 17:58
  • Also you could loop directly from that command that starts with `awk` with a long pipeline using ProcSub. – Jetchisel Jul 11 '21 at 18:02
  • I ran it like this ```while read -r line; do curl -I "$line"; done < <(awk -F "," '{print $12}' "$INPUT" | sed 's,/[^/]*$,,' | sort | uniq)``` and still got the same error. Will try using different fds. – AlexNikolaus Jul 11 '21 at 18:15
  • 1
    It seems like I had carriage returns. Running my command with ```while read -r line; do curl -I "$line"; done < <(awk -F "," '{print $12}' "$INPUT" | sed 's,/[^/]*$,,' | sort | uniq | tr -d "$'\r'")``` worked. I totally forgot to check for this. Thank you very much! – AlexNikolaus Jul 11 '21 at 18:22

0 Answers0