11

I'm new to web programming, and have recently began looking into using Python to automate some manual processes. What I'm trying to do is log into a site, click some drop-down menus to select settings, and run a report.

I've found the acclaimed requests library: http://docs.python-requests.org/en/latest/user/advanced/#request-and-response-objects and have been trying to figure out how to use it.

I've successfully logged in using bpbp's answer on this page: How to use Python to login to a webpage and retrieve cookies for later usage?

My understanding of "clicking" a button is to write a post() command that mimics a click: Python - clicking a javascript button

My question (since I'm new to web programming and this library) is how I would go about pulling the data I need to figure out how I would construct these commands. I've been looking into [RequestObject].headers, .text, etc. Any examples would be great.

As always, thanks for your help!

EDIT::: To make this question more concrete, I'm having trouble interacting with different aspects of a web-page. The following image shows what I'm actually trying to do:

Example

I'm on a web-page that looks like this. There is a drop-down menu with click-able dates that can be changed. My goal is to automate changing the date to the most recent date, "click"'Save and Run', and download the report when it's finished running.

Community
  • 1
  • 1
so13eit
  • 942
  • 3
  • 11
  • 22
  • I haven't had much of a direction to try things with. I did edit the post with a more concrete idea of what I'm trying to do- so I don't think it's just a good parser to extract HTML, but how to "post" different information to the site in order to accomplish different tasks, especially 'clicking' tasks – so13eit Aug 01 '13 at 13:57
  • check out [selenium](https://pypi.python.org/pypi/selenium) if there is javascript in the pages you trying to crawl. – djas Aug 01 '13 at 16:57
  • 1
    One possible workflow for a process like this is to load the page in question (using requests.get()), figure out any dynamic form elements you need to get from the page, and then use requests.post() to submit the information you need. Using a browser's dev tools can be extremely valuable to see exactly what's being send in the POST request. – brechin Oct 10 '13 at 13:24

1 Answers1

8

The only solution to this I have found is Selenium. If it werent a javascript heavy website you could try mechanize but for this you need to render the javascript and then inject javascript...like Selenium does.

Upside: You can record actions in Firefox (using selenium) then export those actions to python. The downside is that this code has to open a browser window to run.

frodopwns
  • 942
  • 9
  • 15