Curl function cannot parse proxy coming from a variable in bash

Question

I have a proxy txt file with the format:

102.129.249.120:3128
102.129.249.120:8080
101.4.136.34:8080
103.228.117.244:8080
etc

and I am trying to create a bash script that will do (for example): curl -x "$IP" google.com.
Unfortunately, curl is giving me an unsupported proxy syntax for all the proxies.
Any ideas?
BTW, I really doubt this question has been repeated as I have tried everything else to no avail.

My script:

Number=$(wc -l < ProxyList.txt)



for ((i=1;i<=$Number;++i))  do
ip=$(head -n ${i} ProxyList.txt | tail -n +${i})
curl -p -x "$ip" 'webpage' -H 'user-agent' -H 'Accept: */*' -H 'Accept-Language: en-US,en;q=0.5' --compressed -H 'DNT: 1' -H 'Connection: keep-alive' -H 'Cookie: wpml_referer_url=referer; _icl_current_language=es; PHPSESSID=tpikve1vl4ued06i082vprqdo1' -H 'If-Modified-Since: Mon, 16 May 2016 07:27:13 GMT' -H 'If-None-Match: "3d6-532f08d9d7640-gzip"' -H 'Cache-Control: max-age=0' -m 6
done

A small sample of my proxy list:

102.129.249.120:3128
102.129.249.120:8080
101.4.136.34:8080
103.228.117.244:8080
103.253.27.108:80
104.45.188.43:3128
104.250.34.179:80
105.27.238.161:80
104.154.143.77:3128
110.243.20.2:9999
111.68.26.237:8080
106.104.151.142:58198
113.252.95.19:8197
115.231.31.130:80
118.69.50.154:80
118.69.50.154:443
119.81.189.194:80
119.81.189.194:8123
119.81.199.81:8123
119.81.199.83:8123
119.81.199.80:8123
12.139.101.100:80
12.139.101.101:80
119.81.199.85:31288
119.81.199.86:8123
119.81.199.87:8123
12.139.101.102:80
124.156.98.172:443
13.228.91.252:3128
138.197.157.32:3128
138.197.157.32:8080
138.68.240.218:8080
138.68.240.218:3128
138.68.60.8:8080
138.68.60.8:3128

of how to use xargs I tried echo $ip | xargs curl url and it still didn't work. Same error. — DAVID, Jun 16 '20 at 17:14
What's the error you get? Also you're missing a semicolon right before the `do` of your `for` loop. — tectux, Jun 16 '20 at 17:52
@tectux Sorry that was a formatting error. I had a space. Error: 'url: (5) Unsupported proxy syntax in '207.154.231.213:8080 — DAVID, Jun 16 '20 at 18:57

tectux · Accepted Answer · 2020-06-16T21:31:22.617

Your input file has carriage return characters at the end of each line.
Each line in your input file ends with \r\n instead of just \n.

You can check with od:

$ head -1 ProxyList.txt | od -c
0000000   1   0   2   .   1   2   9   .   2   4   9   .   1   2   0   :
0000020   3   1   2   8  \r  \n
0000026

So in your script, $ip has in fact a value of 102.129.249.120:3128\r.

You can remove the \r characters with tr for example:

while read proxy; do
  curl -p -x $proxy $webpage
done < <( tr -d '\r' < ProxyList.txt )

score 1 · Answer 2 · answered Jun 16 '20 at 19:13

1

try this:

for ip in $(cat ProxyList.txt)
do
   curl -p -x "$ip" 'webpage' -H 'user-agent' -H 'Accept: */*' -H 'Accept-Language: en-US,en;q=0.5' --compressed -H 'DNT: 1' -H 'Connection: keep-alive' -H 'Cookie: wpml_referer_url=referer; _icl_current_language=es; PHPSESSID=tpikve1vl4ued06i082vprqdo1' -H 'If-Modified-Since: Mon, 16 May 2016 07:27:13 GMT' -H 'If-None-Match: "3d6-532f08d9d7640-gzip"' -H 'Cache-Control: max-age=0' -m 6
done

but the problem with curl might be, that should set the environment variables http_proxy and https_proxy like this:

export http_proxy=http://1.2.3.4:3128/
export https_proxy=http://1.2.3.4:3128/

answered Jun 16 '20 at 19:13

Oliver Gaida

1,722
7
14

Same error ... I'm using ubuntu 20.04 could that be the problem? – DAVID Jun 16 '20 at 19:17
'url: (5) Unsupported proxy syntax in '105.29.64.222:80 – DAVID Jun 16 '20 at 19:19
please give [https://stackoverflow.com/questions/9445489/performing-http-requests-with-curl-using-proxy](https://stackoverflow.com/questions/9445489/performing-http-requests-with-curl-using-proxy) a try – Oliver Gaida Jun 16 '20 at 19:19
first you set the proxy-environment-variables and then call `curl http://www.google.com` etc. – Oliver Gaida Jun 16 '20 at 19:20
Yes. It only works if I input the ip without the variable (manually) – DAVID Jun 16 '20 at 19:22
what is the output of `for ip in $(cat ProxyList.txt);do echo "-$ip-";done` ? – Oliver Gaida Jun 16 '20 at 19:24
That works I have tried that as well it outputs the proxy list as normal! – DAVID Jun 16 '20 at 19:25
@Oliver Gaida, I've seen the http_proxy / https_proxy environment variables work with _wget_, however to my knowledge that does not work with _curl_. – lab9 Jun 16 '20 at 20:19
I tried that out with curl and it "works" (at least it gives me the same error ...) – DAVID Jun 16 '20 at 20:29
Its been solved but thanks! And no I do not know it – DAVID Jun 16 '20 at 21:58
This is the tool together with unix2dos to change the line-endings between dos and unix. – Oliver Gaida Jun 16 '20 at 22:00

lab9 · Answer 3 · 2020-06-16T20:09:19.967

0

As per the curl man page, the -x (or --proxy) switch can be prefixed with the protocol in front of the argument (if omitted, I assume that it defaults to http://):
-x, --proxy [protocol://]host[:port]

A simple bash script with xargs would look like:

#!/bin/bash
webpage=${1:-http://google.com}
cat ProxyList.txt \
| xargs -n1 -I{} curl -p -x http://{} "$webpage" -H 'user-agent' -H 'Accept: */*' -H 'Accept-Language: en-US,en;q=0.5' --compressed -H 'DNT: 1' -H 'Connection: keep-alive' -H 'Cookie: wpml_referer_url=referer; _icl_current_language=es; PHPSESSID=tpikve1vl4ued06i082vprqdo1' -H 'If-Modified-Since: Mon, 16 May 2016 07:27:13 GMT' -H 'If-None-Match: "3d6-532f08d9d7640-gzip"' -H 'Cache-Control: max-age=0' -m 6

edited Jun 16 '20 at 20:09

answered Jun 16 '20 at 20:03

lab9

596
2
8

Still doesn't??? Same error. What version of ubuntu do you have? Or do you have another distro?? I really don't know. – DAVID Jun 16 '20 at 20:07
'url: (5) Unsupported proxy syntax in 'http://46.35.184.187:61003 – DAVID Jun 16 '20 at 20:07
Debian 5.4.19 / curl 7.68.0 – lab9 Jun 16 '20 at 20:10
I'm getting "curl: (56) Received HTTP code 400 from proxy after CONNECT" from your proxies. That seems to be related to your "-p" switch (tunneling). Seems that the proxies don't support that? On the other hand, it shows that the "-x" part of the statements work. – lab9 Jun 16 '20 at 20:14

Curl function cannot parse proxy coming from a variable in bash

3 Answers3