How to use Python to login to a webpage and retrieve cookies for later usage?

Question

I want to download and parse webpage using python, but to access it I need a couple of cookies set. Therefore I need to login over https to the webpage first. The login moment involves sending two POST params (username, password) to /login.php. During the login request I want to retrieve the cookies from the response header and store them so I can use them in the request to download the webpage /data.php.

How would I do this in python (preferably 2.6)? If possible I only want to use builtin modules.

Boden Garman · Answer 1 · 2015-03-16T08:59:16.320

158

Here's a version using the excellent requests library:

from requests import session

payload = {
    'action': 'login',
    'username': USERNAME,
    'password': PASSWORD
}

with session() as c:
    c.post('http://example.com/login.php', data=payload)
    response = c.get('http://example.com/protected_page.php')
    print(response.headers)
    print(response.text)

edited Mar 16 '15 at 08:59

answered Aug 24 '12 at 06:07

Boden Garman

2,455
2
20
17

3

Is `'action': 'login'` actually necessary in this example, or is it just an extra parameter that's sent with the request? – Ted Jun 26 '14 at 16:08
1

@Ted That part is totally needed. – Sanghyun Lee Mar 16 '15 at 08:49
@Ted Maybe it is required in this particular example. It was not needed in my program. – Highstaker Jan 08 '16 at 08:18
This is the best I know of. You might have to data as per website. – Jithin Pavithran Nov 21 '16 at 17:20

Harley Holcombe · Accepted Answer · 2011-09-19T00:40:23.863

146

import urllib, urllib2, cookielib

username = 'myuser'
password = 'mypassword'

cj = cookielib.CookieJar()
opener = urllib2.build_opener(urllib2.HTTPCookieProcessor(cj))
login_data = urllib.urlencode({'username' : username, 'j_password' : password})
opener.open('http://www.example.com/login.php', login_data)
resp = opener.open('http://www.example.com/hiddenpage.php')
print resp.read()

resp.read() is the straight html of the page you want to open, and you can use opener to view any page using your session cookie.

edited Sep 19 '11 at 00:40

answered Oct 09 '08 at 23:24

Harley Holcombe

175,848
15
70
63

1

Is this safe? Won't this allow packet sniffers to see plaintext passwords? Would using Https be more secure? – Chang Hyun Park Dec 10 '12 at 06:58
2

@Heartinpiece Yes, if the server offers it you should use HTTPS. – Harley Holcombe Dec 10 '12 at 22:09
Thank you...but imagin we logged in and want to post sth...how to set coockie in this thread for post data? – MLSC Jun 09 '14 at 10:58
Highly recommend to use requests library if you are writing large code. (personal experience) – swapnil jariwala Aug 29 '15 at 07:05
@HarleyHolcombe what would be the Python 3 equivalent to this code? – mhsnk Dec 09 '20 at 19:51

How to use Python to login to a webpage and retrieve cookies for later usage?

2 Answers2

Linked

Related