Navigating webpages with python/javascript?

Question

I'm working on a program that requires information from some webpages. Currently, I'm getting info by just downloading the source and parsing it (in Python with Regex). However, there's more specific/better data I could be using, but it requires a login and some page navigation by the "user". (The pages with the data can't just be reached by typing in the URL)

I can write javascript/jquery to simulate the logging in and clicking that's needed to navigate to these pages..but currently I can only "run" it by siting in chrome and pulling up the chrome javascript console on each page. Is there a way to simulate browser navigation purely in Python, so I could automate this task? (Also, the page doesn't appear to have any sort of captcha or anti-bot security, o that shouldn't be a problem)

Better you can try **Urllib2**. Ref : http://stackoverflow.com/questions/13925983/login-to-website-using-urllib2-python-2-7. — rash, Nov 12 '14 at 16:50

score 0 · Answer 1 · edited May 23 '17 at 10:33

0

You could check out some browser automation tools, for example Selenium. It's used for website testing primarily, but you can also extract information with it, and then process it in python.

Here is another thread, about using it from python.

edited May 23 '17 at 10:33

Community

1
1

answered Nov 12 '14 at 16:45

zord

4,538
2
25
30

Some response would have been nice. – zord Jan 22 '15 at 10:02

Navigating webpages with python/javascript?

1 Answers1