How to scrape a div AND the contents of an iframe within the div using Selenium and BeautifulSoup?

Question

I'm currently scraping a page formatted as such:

<div id="container>
   <script>Script that cause iframe contents to load correctly</script>
   <iframe>Contents of iFrame</iframe>
   <script>More scripts</script>
</div>

I can scrape the page easily, but this does not scrape the iframe contents, and so I switched frames with:

driver.switch_to.frame(iframeElement)

And this allows me to get the iframe contents. This leads me to my question now of how I can take the container div, and insert the contents of the scraped iframe within the scraped div. The way the page is setup, there are dynamic scripts just before the iframe that allow for the contents of the iframe to work which is why I need to embed the iframe contents within the scraped div.

Relevant Python below:

driver.get(url)
iframeElement = driver.find_element_by_tag_name('iframe')
driver.switch_to.frame(iframeElement)
time.sleep(3) #Wait for the contents to generate
# driver.switch_to_default_content() #Commented out, but I know to use this to exit out of the iframe

html = driver.page_source
soup=BeautifulSoup(html, features="lxml")
print(soup)
print(soup.find("div", {"id": "Container"})) #Let's see the HTML of the container
soupStr=str(soup)
Con = str(soup.find("div", {"id": "Container"})) #Create a variable with JUST the container HTML

with open('iframeWithinDiv.html', 'w', encoding='utf-8') as f_out: #Save the file
    f_out.write(soupStr)```

This might point you in the right direction: https://stackoverflow.com/questions/14088541/how-to-execute-a-javascript-function-in-python-with-selenium — orde, Apr 12 '19 at 17:08
Could you give me an example of how you would like the format to be displayed within the div, with the scraped iframe content :) — David Silveiro, Apr 12 '19 at 18:18
Hey @DavidSilveiro, basically I'm just looking for the contents of the scraped div to be embedded within the div something like this: ``` — JeremyMC27, Apr 12 '19 at 19:30
if you feel like my answer was helpful, when you get a chance, give it a mark :) @JeremyMC27 — David Silveiro, Apr 12 '19 at 21:49

David Silveiro · Answer 1 · 2019-04-12T19:59:52.200

You could append it to the following div by using execute_script and a little bit of jquery (you could use pure JS instead) :)

html = driver.page_source
soup=BeautifulSoup(html, features="lxml")
print(soup)
print(soup.find("div", {"id": "Container"})) #Let's see the HTML of the container
soupStr=str(soup)
Con = str(soup.find("div", {"id": "Container"}))1

#### Append your variable to the given string within wrap ###

driver.execute_script("$('#container').val('newhtmlcontent')")

How to scrape a div AND the contents of an iframe within the div using Selenium and BeautifulSoup?

1 Answers1