How to use proxy with Robobrowser

Question

I'm working with http://robobrowser.readthedocs.org/en/latest/readme.html, (a new python library based on the beautiful soup and request libraries) within django. My django app contains :

def index(request):    

    p=str(request.POST.get('p', False)) # p='https://www.yahoo.com/'

    pr="http://10.10.1.10:3128/"
    setProxy(pr)

    browser = RoboBrowser(history=True)
    postedmessage = browser.open(p)
    return HttpResponse(postedmessage)

I would like to add a proxy to my code but can't find a reference in the docs on how to do this. Is it possible to do this?

EDIT:

following your recommendation I've changed the code to

    pr="http://10.10.1.10:3128/"
    setProxy(pr)
    browser = RoboBrowser(history=True)

with:

def setProxy(pr):
    import os
    os.environ['HTTP_PROXY'] = pr
    return

I'm now getting:

Django Version: 1.6.4
Exception Type: LocationParseError
Exception Value:    
Failed to parse: Failed to parse: 10.10.1.10:3128

Any ideas on what to do next? I can't find a reference to this error

score 10 · Answer 1 · answered Jul 28 '14 at 23:34

After some recent API cleanup in RoboBrowser, there are now two relatively straightforward ways to control proxies. First, you can configure proxies in your requests session, and then pass that session to your browser. This will apply your proxies to all requests made through the browser.

from requests import Session
from robobrowser import RoboBrowser

session = Session()
session.proxies = {'http': 'http://my.proxy.com/'}
browser = RoboBrowser(session=session)

Second, you can set proxies on a per-request basis. The open, follow_link, and submit_form methods of RoboBrowser now accept keyword arguments for requests.Session.send. For example:

browser.open('http://stackoverflow.com/', proxies={'http': 'http://your.proxy.com'})

I just realized that robobrowser is your project. Thank you for creating it! - Bill — user1592380, Jul 29 '14 at 21:01
BTW, if you have a second, I have a new question at http://stackoverflow.com/questions/25067874/how-to-submit-a-form-with-a-get-action-using-robobrowser — user1592380, Aug 01 '14 at 15:06

score 2 · Accepted Answer · edited Sep 16 '20 at 08:43

2

Since RoboBrowser uses the request library, you can try to set the proxies as mentioned in the request docs by setting the environment variables HTTP_PROXY and HTTPS_PROXY.

edited Sep 16 '20 at 08:43

Nitish

565
4
16

answered Jun 12 '14 at 06:25

arocks

2,862
1
12
20

Try using "10.10.1.10:3128" as proxy value or "http://user:pass@10.10.1.10:3128/" – arocks Jun 12 '14 at 18:13
It turned out that I had a few additional charachters at the end of the proxy string. when I removed them it started working using pr="http://10.10.1.10:3128/". Thanks very much for the help. - Bill – user1592380 Jun 12 '14 at 23:52

How to use proxy with Robobrowser

2 Answers2

Linked