0

I'm looking to scrape the contents of a page that requires you to press an arrow button in which, information will appear via jquery rather than loading a new page. Since there needs to be a button click, I'm using mechanize for this part instead of nokogiri. What I have so far is 

url = "http://brokercheck.finra.org/Individual/Summary/1327992"
mechanize = Mechanize.new 
page = mechanize.get(url)
button = page.at('.ArrowExpandDsclsr.faangledown')
new_page = mechanize.click(button)
new_page.at('#disclosuredetails')

It appears that new_page still doesn't show the page with the newly loaded information. Anyone know why that is?

chonglawr
  • 201
  • 1
  • 3
  • 15
  • 2
    Mechanize does not process Javascript. Have a look at the answers for http://stackoverflow.com/questions/802225/how-do-i-use-mechanize-to-process-javascript – eugen Apr 11 '16 at 21:29

1 Answers1

0

The button you're trying to get mechanize to click is not a "regular" button, it's a bit more dynamic. It uses javascript / ajax to fetch the relevant data when clicked.

Mechanize doesn't render the DOM of a web page nor it provides a way to have javascript interact with the page. Thus it's not suitable for interacting with dynamic pages depending on javascript for their functionality.

For such cases, I'd suggest phantomjs, either standalone or through capybara / poltergeist if you'd prefer interacting with it via ruby.

egwspiti
  • 957
  • 5
  • 10