1

Have to write a Python app to communicate and fetch some data from a password protected, JavaScript enabled web app.

The webapp unfortunately requires JavaScript to run, and refuses to work if JavaScript is disabled. Therefore, 'urllib2' and 'requests' don't work when trying to fetch data.

What are my options at this point? If I grabbed HTTP traffic via some tool such as Firebug, and tried re-playing this traffic via some Python method, would this work? All I need for the script to do is:

  1. Login into website
  2. Change a dropdown item
  3. Grab a single line of text and logout
halfer
  • 19,824
  • 17
  • 99
  • 186
Cmag
  • 14,946
  • 25
  • 89
  • 140
  • 1
    Well, yes. The server has no way of knowing if the request was made by the JavaScript or by other code. You'd have to reverse engineer the requests made by the webapp, and reproduce them in Python. Firebug would be a good starting point, maybe try Wireshark too. – cha0site Jan 19 '12 at 15:38
  • 1
    This related [question](http://stackoverflow.com/q/8323728/183066) should be useful to your problem. – jcollado Jan 19 '12 at 16:00
  • Awesome :) Thanks guys! If you can point me to a good article on how to replay http packets... that would be awesome :))))) Will keep rtfming – Cmag Jan 19 '12 at 16:29

1 Answers1

1

Your best bet is to run a headless browser like PyPhantomJS which understands javascript. You have to write your code in javascript and than you can do whatever you want.

It has fast and native support for DOM handling, CSS selector and of course JavaScript.

RanRag
  • 48,359
  • 38
  • 114
  • 167
  • ABSOLUTELY! Thank you! I just started learning javascript as well! This is turning into a super fun project :) – Cmag Jan 19 '12 at 17:06
  • @RanRag... hitting a wall here.. i can get phantomjs to execute scripts... but not actually log into the website and pull down some data.... – Cmag Jan 19 '12 at 18:31
  • I found this: http://code.google.com/p/phantomjs/issues/attachmentText?id=295&aid=2950000000&name=test.js&token=y_nvU37fSckqtZbyW5rSAtQ_rrQ%3A1326997938627... but it still isnt logging me into my web app – Cmag Jan 19 '12 at 18:36
  • 1
    Try using firefox's `live http header` addon and see what all request are made when you manually try to login. – RanRag Jan 19 '12 at 18:38
  • May be your website require cookies to be send by the user. – RanRag Jan 19 '12 at 18:38