Selenium Python: clicking links produced by JSON application

Question

[ Ed: Maybe I'm just asking this? Not sure -- Capture JSON response through Selenium ]

I'm trying to use Selenium (Python) to navigate via hyperlinks to pages in a web database. One page returns a table with hyperlinks that I want Selenium to follow. But the links do not appear in the page's source. The only html that corresponds to the table of interest is a tag indicating that the site is pulling results from a facet search. Within the div is a <script type="application/json"> tag and a handful of search options. Nothing else.

Again, I can view the hyperlinks in Firefox, but not using "View Page Source" or Selenium's selenium.webdriver.Firefox().page_source call. Instead, that call outputs not the <script> tag but a series of <div> tags that appear to define the results' format.

Is Selenium unable to navigate output from JSON applications? Or is there another way to capture the output of such applications? Thanks, and apologies for the lack of code/reproducibility.

Answer 1

Try using execute_script() and get the links by running JavaScript, something like:

driver.execute_script("document.querySelector('div#your-link-to-follow').click();")

Note: if the div are generated by scripts dynamically, you may want to implicitly wait a few seconds before executing the script.

Answer 2

I've confronted a similar situation on a website with JavaScript (http://ledextract.ces.census.gov to be specific). I had pretty good luck just using Selenium's get_element() methods. The key is that even if not everything about the hyperlinks appears in the page's source, Selenium will usually be able to find it by navigating to the website since doing that will engage the JavaScript that produces the additional links.

Thus, for example, you could try mousing over the links, finding their titles, and then using:

driver.find_element_by_xpath("//*[@title='Link Title']").click()

Based on whatever title appears by the link when you mouse over it.

Or, you may be able to find the links based on the text that appears on them:

driver.find_element_by_partial_link_text('Link Text').click()

Or, if you have a sense of the id for the links, you could use:

driver.find_element_by_id('Link_ID').click()

If you are at a loss for what the text, title, ID, etc. would be for the links you want, a somewhat blunt response is to try to pull the id, text, and title for every element off the website and then save that to a file that you can look for to identify likely candidates for the links you're wanting. That should show you a lot more (in some respects) than just the source code for the site would:

AllElements = driver.find_elements_by_xpath('//*')
for Element in AllElements:
    print 'ID = %s   TEXT = %s   Title =%s' %(Element.get_attribute("id"), Element.get_attribute("text"), Element.get_attribute("title"))

Note: if you have or suspect you have a situation where you'll have multiple links with the same title/text, etc. then you may want to use the find_elements (plural) methods to get lists of all those satisfying your criteria, specify the xpath more explicitly, etc.