123

I am trying to log in to a page and access another link in the page.

I get a "405 Not Allowed" error from this attempt:

payload={'username'=<username>,'password'=<password>}
with session() as s:
    r = c.post(<URL>, data=payload)
    print(r)
    print(r.content)

I checked the post method details using Chrome developer tools and found a URL that appeard to be an API endpoint. I posted to that URL with the payload and it seemed to work; I got a response similar to what I could see in the developer.

Unfortunately, when trying to 'get' another URL after logging in, I am still getting the content from the login page. Why is the login not sticking? Should I use cookies? How?

Karl Knechtel
  • 62,466
  • 11
  • 102
  • 153
user1474157
  • 1,379
  • 2
  • 11
  • 13

4 Answers4

162

You can use a session object. It stores the cookies so you can make requests, and it handles the cookies for you

s = requests.Session() 
# all cookies received will be stored in the session object

s.post('http://www...',data=payload)
s.get('http://www...')

Docs: https://requests.readthedocs.io/en/master/user/advanced/#session-objects

You can also save the cookie data to an external file, and then reload them to keep session persistent without having to login every time you run the script:

How to save requests (python) cookies to a file?

boltronics
  • 35
  • 6
gtalarico
  • 4,409
  • 1
  • 20
  • 42
  • 3
    :O This is amazing! I wish I had known about this feature before. Is it storing the cookies on disk like a browser, or just holding them in memory? – Aurelius Jun 30 '18 at 10:16
  • 3
    It just keeps them in memory while the application is running. To save/load cookies from disk see this: https://stackoverflow.com/a/30441145/4411196 – gtalarico Jul 20 '18 at 13:51
  • I instead of saving them, made a global variable assigned it a session and made a function so it logins if the cookie is not there and/or returns the session – yashas123 Jul 13 '20 at 15:06
  • for me print(s.cookies) shows that cookies are in the session but are not always passed in subsequent requests (in fact the same code works against localhost but against a test server does not) – Andrzej Martyna Dec 30 '21 at 10:55
97

From the documentation:

  1. get cookie from response

     url = 'http://example.com/some/cookie/setting/url'
 r = requests.get(url)
 r.cookies

    {'example_cookie_name': 'example_cookie_value'}

  2. give cookie back to server on subsequent request

     url = 'http://httpbin.org/cookies'
 cookies = {'cookies_are': 'working'}
 r = requests.get(url, cookies=cookies)`
James Wong
  • 83
  • 1
  • 6
Freek Wiekmeijer
  • 4,556
  • 30
  • 37
  • Thanks. Actually, it seems there are no cookies that getting created. I checked the request headers and could not see any 'cookies'. Meanwhile there is one cookie created in response headers. How can i make my login stick if there is no cookie. – user1474157 Jul 22 '15 at 09:40
  • 2
    The normal flow for an authentication cookie is: (1) when you submit a login form, you receive a cookie in the response headers. (2) on subsequent page requests, you add the cookie to the request headers. – Freek Wiekmeijer Jul 22 '15 at 09:54
  • how do you add multiple cookies ? – Jitin Oct 27 '20 at 06:11
  • 2
    The kwarg “cookies” is a dict, you can add as many items as you want. – Freek Wiekmeijer Oct 27 '20 at 06:38
19

Summary (@Freek Wiekmeijer, @gtalarico) other's answer:

Logic of Login

  • Many resource(pages, api) need authentication, then can access, otherwise 405 Not Allowed
  • Common authentication=grant access method are:
    • cookie
    • auth header
      • Basic xxx
      • Authorization xxx

How use cookie in requests to auth

  1. first get/generate cookie
  2. send cookie for following request
  • manual set cookie in headers
  • auto process cookie by requests's
    • session to auto manage cookies
    • response.cookies to manually set cookies

use requests's session auto manage cookies

curSession = requests.Session() 
# all cookies received will be stored in the session object

payload={'username': "yourName",'password': "yourPassword"}
curSession.post(firstUrl, data=payload)
# internally return your expected cookies, can use for following auth

# internally use previously generated cookies, can access the resources
curSession.get(secondUrl)

curSession.get(thirdUrl)

manually control requests's response.cookies

payload={'username': "yourName",'password': "yourPassword"}
resp1 = requests.post(firstUrl, data=payload)

# manually pass previously returned cookies into following request
resp2 = requests.get(secondUrl, cookies= resp1.cookies)

resp3 = requests.get(thirdUrl, cookies= resp2.cookies)
crifan
  • 12,947
  • 1
  • 71
  • 56
2

As others noted, Here is an example of how to add cookies as string variable to the headers parameter -

    headers = {
        'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) ...',
        'cookie': '_fbp=fb.1.1654447470850.2143140577; _ga=GA1.2.1...'
    }
    response = requests.get(url, headers=headers)
Matan Dobrushin
  • 107
  • 1
  • 10