1

This is a private website, so I will try my best to explain it.

Scraping with Selenium, and table is not in the HTML at all, although it is visible to the user and interactable.

The HTML code on the page is accessed through a navbar click, and then the HTML is pretty simple. Has three or so divs, and some scripts.

<head></head>
<body>
 <div></div>
 <div></div>
</body>

But the actual webpage has an entire dynamic table that opens reports in new windows when clicked on as well as a bunch of other stuff.

Why can I see everything on the webpage, but not in Selenium OR with Inspect Element? (unless I inspect search bar)

The missing HTML showed up when I clicked inspect element on the search bar, and from there I was able to view the HTML, but that isn't a solution for Selenium.

I dont know if this helps, but the ID's of the hidden HTML have this in it: 'crmGrid_visualizationCompositeControl'

Thank you!

Matthew Rozanoff
  • 95
  • 6
  • What happens when you do F12 and inspect the elements of the table? Can you provide class or id of the element you are trying to find with a screenshot? – Tal Angel Sep 12 '22 at 15:30
  • @TalAngel F12 does not return the table HTML. While the table exists on the page, it does not show up in the developers console, that is whats getting me stuck – Matthew Rozanoff Sep 12 '22 at 16:45
  • So - is the table an image? If it's a HTML element you must be able to inspect it. Please share the link to the site – Tal Angel Sep 12 '22 at 16:49
  • You won't be able to access it without a login. That is what I cannot understand, I can only see the table with my eyes, it is not there on the HTML **unless** I inspect element on the search bar. Just F12 does not show the table – Matthew Rozanoff Sep 12 '22 at 17:07
  • Is that what you need? https://stackoverflow.com/questions/15370838/inspect-hovered-element-in-chrome – Tal Angel Sep 12 '22 at 17:27
  • I don't think so, f8 didn't change much. Im trying to think of a better way to explain but Im not sure how. – Matthew Rozanoff Sep 12 '22 at 17:43
  • @TalAngel It's an iframe, does that mean anything? The div that contains the iframe does not appear in f12 until I inspect something inside the iframe area. Trying to brainstorm a way to trigger the iframe with selenium... – Matthew Rozanoff Sep 13 '22 at 15:34

1 Answers1

1

Makes sense, Selenium does NOT know how to find elements inside an Iframe. You must tell the Selenium to switch into the Iframe so it will be able to "see" the elements inside:

iframe = driver.find_element_by_xpath("//iframe[@name='Name_Of_Iframe']")
driver.switch_to.frame(iframe)

If the Iframe has an Id - find it by Id. You can also switch to other iframes by index:

iframe = driver.find_elements_by_tag_name('iframe')[2]
driver.switch_to.frame(iframe)
Tal Angel
  • 1,301
  • 3
  • 29
  • 63