Python reading url that requires logging in

Question

I use

player_id = re.compile('<td><a href="(.+?)" onclick=')
url = 'http://www.csfbl.com/freeagents.asp?leagueid=2237'
sock = urllib.request.urlopen(url).read().decode("utf-8")

or

import pandas as pd
pd.read_html(url)

The website in question displays two different values based on whether I am logged in or not. How can I get the values as if I was logged in?

http://www.csfbl.com

is the website.

Thanks in advance!

See: http://stackoverflow.com/questions/2910221/how-can-i-login-to-a-website-with-python — jmunsch, Jun 03 '15 at 02:31
i'm like... a newbie, i don't understand what i need to do... can you explain? — Euisoo Yoo, Jun 03 '15 at 02:35
I logged in on chrome / explorer before running my code, it always returns the values as if I was not logged in... help? — Euisoo Yoo, Jun 03 '15 at 02:42
I should have been more clear. If you want python to use the site as if you were logged in, then you would want to login using python. — jmunsch, Jun 03 '15 at 02:49

score 2 · Answer 1 · answered Jun 03 '15 at 02:55

Use the requests module to submit the credentials on the web-form at: http://www.csfbl.com/logon.asp .

Instructions for submitting web forms using requests can be found here: http://docs.python-requests.org/en/latest/user/quickstart/#more-complicated-post-requests

Other sites can be more challenging. For those, consider using selenium or mechanize.

Python reading url that requires logging in

1 Answers1