Proxies from a file with Requests

Question

How do I use proxies from a text file with requests? The proxies in the file are in the format: ip:port So I can't really build a dict like {'scheme' : 'scheme://ip:port'}

And I need to access two different sites with the same proxy and then switch the proxy. One of the site uses HTTP and the other uses HTTPS.

I tried doing this for the HTTP request:

response = c.get(url, proxies = {'http': 'http://'+p})

And this for the HTTPS request:

response = c.get(url, proxies = {'https': 'https://'+p})

But the first one doesn't work and throws me an error.

Any workarounds for this?

*“So I can't really build a dict”* – Why not? Just parse the file and build a dictionary. *“But the first one doesn't work and throws me an error”* – What is the full error, what exactly are you doing, what is the value of `p`, etc. You need to provide more information than that if you want us to help you. — poke, Jun 01 '15 at 06:21
The page doesn't load. I mean I check for a particular cookie and it is assigned to NoneType. — Praneeth, Jun 01 '15 at 06:28

score 1 · Answer 1 · edited May 23 '17 at 10:26

My assumption is that it fails due to an untrusted SSL certificate.

requests.exceptions.SSLError: [Errno 1] _ssl.c:503: error:14090086:SSL routines:SSL3_GET_SERVER_CERTIFICATE:certificate verify failed

To solve this, read the following post

However, if you get any similar message to this

Connection to server refused.

It can be caused by many reasons, such as firewall blocking the port or proxy is configured badly. Anyway, to see if that is a bug in your program, try to use the proxy (SSL) in your browser and surf the internet.

Regarding your second question, that shouldn't be a problem. For instance, the file proxies.txt has the following data:

172.17.0.3:443
172.17.0.23:9443
172.17.0.34:80

Then, you can infer the scheme according to the port number. It is very common that ports that end with 443 are HTTPS.

with open('proxies.txt', 'r') as data
    lines = data.readlines()

proxies = {'http': [], 'https': []}
for line in lines:
    ip_port = line.split(':')
    ip, port = (ip_port[0], ip_port[1]) if len(ip_port) > 1 else (ip_port[0], 80)
    scheme = 'http'
    if port.endswith('443'):
        scheme = 'https'
    proxies[scheme].append(join([scheme, '://', ip, ':', port]))

The page doesn't load. I mean I check for a particular cookie and it is assigned to NoneType. — Praneeth, Jun 01 '15 at 06:31
what do you mean the `cookie` is assigned to `None`? You don't get any error message? What is the response's `status_code` and `headers`, or anything that gives more information about the problem. — boaz_shuster, Jun 01 '15 at 06:36

Proxies from a file with Requests

1 Answers1