Using urllib.request returns Proxy Auto-Config file

Question

I am using Martin Konecny's code from here to query an http site, from behind my corporate firewall:

The code is this:

    import urllib.request
req = urllib.request.Request(
    'http://www.espncricinfo.com/', 
    data=None, 
    headers={
        'User-Agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_3) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/35.0.1916.47 Safari/537.36'
    }
)

f = urllib.request.urlopen(req)
g = open('writing.txt','w')
g.write(f.read().decode('utf-8'))
g.close

However, once I run this code, I receive the PAC file and not the contents of the url.

How do I get past it to download the contents of the website as given the url?

Thank you!

What exactly you want to download ?? using g.write(f.read().decode('utf-8')) you are write the full response body into writing.txt file — ergesto, Jul 09 '18 at 12:17
@ergesto I am receiving the PAC file as response and not the contents of the URL. Do note, I am operating behind a corporate firewall. — Soham, Jul 09 '18 at 12:48
Hi, @ergesto many many thanks. This has solved the problem. Can you please post this as an answer so that I can vote it upwards and call it the solution? — Soham, Jul 09 '18 at 13:42

score 2 · Accepted Answer · answered Jul 09 '18 at 15:23

import urllib.request

req = urllib.request.Request('http://www.espncricinfo.com/', data=None, headers={
    'User-Agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_3) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/35.0.1916.47 Safari/537.36'
    }
)

proxy_support = urllib.request.ProxyHandler({'http': 'ip:port'})
opener = urllib.request.build_opener(proxy_support)
# make opener object the global default opener. 
urllib.request.install_opener(opener)


f = urllib.request.urlopen(req)

g = open('writing.txt','w')
g.write(f.read().decode('utf-8'))
g.close

Using urllib.request returns Proxy Auto-Config file

1 Answers1