2

I'm using mechanize to visit the same web-site from different proxies. The website has a log-in page. I have 5 proxies and 5 different log-ins, one to use for each proxy.

If I just run my script with one proxy and one log-in, each one works fine. However, if I run two or more proxies/log-ins at once, then I start getting errors (from the website) like "either your session has timed out or cookies are not enabled." This happens whether I'm running the 5 instances from the same script (same process), or from different scripts (different processes).

What would cause this to work individually but not all at once?

Claudiu
  • 224,032
  • 165
  • 485
  • 680

1 Answers1

2

That is because mechanize automatically creates a shared "cookie jar" by default. For more advanced cookie handling options, you will have to create your own cookie jar for each of the script sessions.

I had to use a custom cookie jar in a past project, in order to move cookies from one session to another. The end result is the same, each instance of your script would have it's own unique file to store it's cookies, so it's on you to manage the cookie files and know that they don't get confused.

>>>> import mechanize

>>>> cj1 = mechanize.CookieJar()
>>>> cj2 = mechanize.CookieJar()
>>>> mech1 = mechanize.OpenerFactory().build_opener(mechanize.HTTPCookieProcessor(cj1))
>>>> mech2 = mechanize.OpenerFactory().build_opener(mechanize.HTTPCookieProcessor(cj2))

>>>> request = mechanize.Request('http://example.com') # testing shows they can share a request

>>>> response1 = mech1.open(request)
>>>> response2 = mech2.open(request)

>>>> print cj1
<mechanize._clientcookie.CookieJar[<Cookie JSESSIONID=54FBB2BE99E4CFDA8F8386F52FCF59C3>]>
>>>> print cj2
<mechanize._clientcookie.CookieJar[<Cookie JSESSIONID=350C0D544CDAD344A1272DA8D7B016B0>]>

In this example I've tested, you can see the two mechanize objects, each with it's own independent cookie jar.

Simon Hova
  • 293
  • 2
  • 12
  • Even if the scripts are running in different processes, they share a cookie jar? I also did the following with my scripts: " ` self.cj = mechanize.CookieJar(); self.mech = mechanize.Browser(factory=mechanize.RobustFactory()); self.mech.set_cookiejar(self.cj)`", and the same behavior happened. I'll try it your way, as well, but is there a reason my way didn't work (either the cookie jar or hte scripts?) – Claudiu Feb 13 '12 at 22:30
  • I copied my code a bit too verbatim, and left out a bit at the end. I've modified and tested, and it works now. See the update above. – Simon Hova Feb 14 '12 at 14:41
  • How would I use this with the `mechanize.Browser()`? My code already uses the browser and I'd rather not change it, if possible. – Claudiu Feb 14 '12 at 15:15
  • Looking over the source code, and it doesn't look good. The Browser class abstracts the cookie jar entirely, which means that it's locked in there pretty good, and doesn't allow outside users to mess with it. The good part is that it looks like it's only mentioned in one or two places, so you may be able to fork the source code, and make a version of mechanize that could show the cookie jar, but it would be far more work than changing your code to stop using the Browser class. – Simon Hova Feb 14 '12 at 15:56
  • hmm I do see a `set_cookiejar` function but it doesn't seem to do anything.. that is, i set it to two different cookie jars, but they seem to interfere w/ each other. will try it your way and see what happens – Claudiu Feb 14 '12 at 17:00
  • hm well actually the cookie jars do seeem separate but i'm still getting the error... might be something else (especially considering that, with **different processes**, the same problem manifests). i think your answer answers this question, though. do you have any insight to offer into what else it might be? – Claudiu Feb 14 '12 at 17:12
  • It's tough to diagnose further without seeing your source code, or even better some Fiddler2 logs. I can say that I am using this [code](http://stackoverflow.com/questions/6111372/multi-threaded-web-scraper-using-urlretrieve-on-a-cookie-enabled-site/6125654#6125654) (or a future revision of it) to log into a website using four different threads, and I don't have a problem with the cookies. – Simon Hova Feb 14 '12 at 18:04