automate navigation of a JavaScript powered website

Question

I need to automate navigation around a JavaScript powered website in Python so I can scrape some content. I came across Chickenfoot, which is a FireFox extension that gives me a programming interface to the browser.

Do you know of other solutions?

Answer 1

WWW::Mechanize has several extensions/compatible replacements to handle JavaScript: WWW::Mechanize::FireFox, WWW::Mechanize::Plugin::JavaScript/WWW::Scripter::Plugin::JavaScript, Mozilla::Mechanize, Gtk2::WebKit::Mechanize, Win32::IE::Mechanize.

Answer 2

Check out Selenium.

Answer 3

You can automate Internet Explorer pretty easily from either javascript (.js files) or any other language that can use COM (c#, perl etc)

http://msdn.microsoft.com/en-us/library/aa752084%28VS.85%29.aspx

Simple example in vbscript:

Dim objIE
Dim objWebForm
Dim objDoc
dim leCount
dim objElement
dim objElementCollection
dim leIndex

Set objIE = WScript.CreateObject("InternetExplorer.Application")
objIE.AddressBar = true
objIE.Visible = true

Sub WaitForLoad (objIE)
  Do While objIE.Busy
    WScript.Sleep(1000)
  Loop
  WScript.Sleep(500)
End Sub

objIE.Navigate("http://www.softtesting.org/")
WaitForLoad(objIE)
set objDoc = objIE.document

Set objElementCollection = objDoc.getElementsByTagName("a")
leCount = objElementCollection.length
For leIndex = 0 To leCount-1
    Set objElement =  objElementCollection(leIndex)
    If (("http://www.softtesting.org/forum/")=objElement.href) Then
        objElement.Click()
    End If
Next

http://social.msdn.microsoft.com/Forums/en-US/windowsaccessibilityandautomation/thread/6c47aaec-6beb-4b21-95b2-95186f5bb4a5

Answer 4

check http://watin.org/ it's like chickenfoot, but it also support ie. And it's free.