1

I'm building an extra provider for Sickbeard and having problems with my cookies. I've been looking for a long time now after why cookies are missing in the HTTP Response when using requests.

login_params = {'uid': sickbeard.PROVIDER_USERNAME,
                'pwd': sickbeard.PROVIDER_PASSWORD,
               }

try:
    response = self.session.post(self.urls['login'], data=login_params, timeout=30)
except (requests.exceptions.ConnectionError, requests.exceptions.HTTPError), e:
    logger.log(u'Unable to connect to ' + self.name + ' provider: ' +ex(e), logger.ERROR)
    return False


The response only contains one cookie the __cdfuid cookie:

requests.utils.dict_from_cookiejar(self.session.cookies)['__cfduid']


The cookies that I get (and want) when logging into the provider are __cdfuid | uid | pass:

requests.utils.dict_from_cookiejar(self.session.cookies)['__cfduid']
requests.utils.dict_from_cookiejar(self.session.cookies)['uid'] #Not passed
requests.utils.dict_from_cookiejar(self.session.cookies)['pass'] #Not passed


I don't know if it matters but the __cdfuid cookie is the only one that has HttpOnly and path=/ parameters set. The other two just has expiration and the actual data.

Philip
  • 137
  • 1
  • 2
  • 16
  • what is `response.cookies`? – jfs Apr 13 '14 at 00:39
  • Only one param, **__cfduid** not the other two. – Philip Apr 13 '14 at 00:44
  • how do you know that `uid` and `pass` cookies are set? – jfs Apr 13 '14 at 00:47
  • When I look into the `response.text` I see that I successfully logged in, so I presume the cookies to be set. When I try the same post request in Chrome I get the header with the `uid` and `pass` cookies set. – Philip Apr 13 '14 at 00:51
  • successful login doesn't mean that `uid` and `pass` must be set. What happens if you clear cookies in Chrome before the request? – jfs Apr 13 '14 at 00:55
  • Same result, all the cookies are set as they should. – Philip Apr 13 '14 at 00:58
  • do the subsequent requests using `self.session` succeed? – jfs Apr 13 '14 at 01:00
  • No, that's the thing. The upcoming requests using `self.session` _(i.e. searching the provider)_ requires the cookies to be set. But because I can't get the complete set of cookies in the response, the cookies are not set, so hence the request fails. – Philip Apr 13 '14 at 01:05
  • you could use a network sniffer e.g., wireshark: to see the difference between Chrome and `requests` in this case. – jfs Apr 13 '14 at 01:20
  • This won't work, unfortunately. All traffic is encrypted through `HTTPS` requests. Maybe this has something to do with the problem? – Philip Apr 13 '14 at 02:07
  • Use any means that are convenient for you to see request/responses (browser tools, debug flags in `requests`, sslsniff). Do you see `secure` in the `Set-Cookie` header? Is `self.urls['login']` an https url? – jfs Apr 13 '14 at 02:30
  • No the `Set-Cookie` header does not contain any `secure` params. I can see that the request sent out from python is a `HTTP/1.0` and the one with chrome is `HTTP/1.1`. How would I go on by forcing the `self.session.post` to send a `HTTP/1.1` request instead? – Philip Apr 13 '14 at 09:51
  • `requests` is HTTP/1.1 client. If you see `HTTP/1.0` something is wrong. For debugging, try [`urllib2` client](http://stackoverflow.com/a/8206372/4279) – jfs Apr 13 '14 at 10:34
  • Ok, tried the same with `urllib2` and got the same results. It send out a `HTTP/1.0` request. Then I tried sending a `POST /takelogin.php HTTP/1.0` request using **cURL** _(Through the terminal)_ and got the cookies I wanted.. I don't get this? – Philip Apr 13 '14 at 11:02
  • Never mind my last comment. I ran the debugging they listed here [link](http://stackoverflow.com/a/16630836/1052393) and got all information and it seems like it sends a `HTTP/1.1` with a `302 found` response. All the headers looks correct BUT I only get one cookie `__gfduid`. – Philip Apr 13 '14 at 11:13
  • So, I have no clue of what to do. Do you think there is something wrong with the host? – Philip Apr 13 '14 at 18:46
  • Try selenium webdriver or ghost.py to send the form and see whether you get the cookies – jfs Apr 13 '14 at 19:06

1 Answers1

3

response.cookies contains only the cookies set by that response. If you are redirected you may find some cookies were set by the redirect. These will not be present on response.cookies, though they will be present on response.history[i].cookies.

You should always examine the cookies dictionary on the Session, not the responses, if you want to get a full idea of what cookies have been set.

Lukasa
  • 14,599
  • 4
  • 32
  • 34
  • Yeah, but how is it that the I posted above uses `self.session` to get the cookies and it only returns the `__cfduid` cookie? – Philip Apr 19 '14 at 23:10
  • Check all of the history responses and confirm that the cookies were in fact set. – Lukasa Apr 21 '14 at 08:03