How to route urllib requests through the TOR network?

Question

What have you tried? TOR should be largely transparent to you. Try using urrlib2; post your code and error messages. — S.Lott, Apr 02 '09 at 19:59
I have no code or error messages - I am asking how to do it. — Lobe, Apr 02 '09 at 20:04
@Lobe: Tor anonymizes your requests -- it conceals you from the web site. It doesn't do anything to the basic method of making HTTP requests -- that's why there's no documentation. Nothing changes except no you're anonymous. — S.Lott, Apr 02 '09 at 20:24

score 12 · Answer 1 · answered Jul 24 '10 at 01:04

12

This works for me (using urllib2, haven't tried urllib):

def req(url):
    proxy_support = urllib2.ProxyHandler({"http" : "127.0.0.1:8118"})
    opener = urllib2.build_opener(proxy_support) 
    opener.addheaders = [('User-agent', 'Mozilla/5.0')]
    return opener.open(url).read()

print req('http://google.com')

answered Jul 24 '10 at 01:04

jahmax

8,181
7
26
25

2

http://stackoverflow.com/questions/1096379/tor-with-python/2015649#2015649 – Dmitri Farkov Mar 25 '11 at 18:10

score 6 · Answer 2 · answered Apr 02 '09 at 20:15

6

Tor works as a proxy, right? So ask yourself "How do I use proxies in urllib?"

Now, when I look at the docs, first thing I see is

urllib.urlopen(url[, data[, proxies]])

which seems pretty suggestive to me...

answered Apr 02 '09 at 20:15

dmckee --- ex-moderator kitten

98,632
24
142
234

I tried that, didnt work: >>> urllib.urlopen('http://www.google.com',proxies={'http':'http://127.0.0.1:9051'}) > >>> _.read() – Lobe Apr 03 '09 at 21:30
But Tor works for you from other apps? Can you use _any_ proxies with liburl? – dmckee --- ex-moderator kitten Apr 04 '09 at 02:15
Doesn't tor works on 127.0.0.1:8118?, I think 9051 is the control port. – jahmax Jul 24 '10 at 01:03
No, TOR runs on 9050 by default, 9051 is control, and 8118 is for some extra privacy-helping proxies (edit: privoxy I think). – Thomas M. DuBuisson Jul 24 '10 at 01:16
1

Oh yea, privoxy and polipo uses 8118 as default port. – jahmax Jul 24 '10 at 01:21

score 2 · Answer 3 · edited May 23 '17 at 12:17

I managed to do an urlib.request for an onion url I found a solution based on this post: Python 3.2 : urllib, SSL and TOR through socket : error with fileno function

here is the modified code:

import socks
import socket

# This function has no DNS resolve
# it need to use the real ip adress to connect instead of www.google.com
def create_connection_fixed_dns_leak(address, timeout=None, source_address=None):
    sock = socks.socksocket()
    sock.connect(address)
    return sock

# MUST BE SET BEFORE IMPORTING URLLIB
socks.setdefaultproxy(socks.PROXY_TYPE_SOCKS5, "127.0.0.1", 9050)
# patch the socket module
socket.socket = socks.socksocket
socket.create_connection = create_connection_fixed_dns_leak

from urllib import request

if __name__ == "__main__":
    for proxy in request.getproxies():
        print(str(proxy))
    url = 'http://url_of_hidden_service.onion:port'
    req = request.Request(url)
    res = request.urlopen(req)
    print(str(res.read()))

How to route urllib requests through the TOR network?

3 Answers3

Linked

Related