Incomplete HTML-response on some sites using Requests & BeautifulSoup or Selenium

Question

I'm tying to scrape information from some urls using Requests and BeautifulSoup in Python. But some sites only return an partial HTML response missing the content of the page

This is the code, that is not working:

import requests
from bs4 import BeautifulSoup
url = "http://www.exampleurl.com"
r = requests.get(url)
soup = BeautifulSoup(r.content, 'html.parser')

Here is the incomplete response: Picture

I tried to use Selenium with Chrome Webdriver instead, but ended up with the same issue.

from selenium import webdriver
options = webdriver.ChromeOptions()
options.add_argument('--ignore-certificate-errors')
options.add_argument('--incognito')
options.add_argument('--headless')
browser = webdriver.Chrome(options=options)
browser.get(url)
html = browser.page_source

Any ideas?

Please see this https://stackoverflow.com/questions/45246280/selenium-drivers-page-source-different-than-browser — Kishan Yadav, Jan 09 '21 at 09:02

score 1 · Accepted Answer · answered Jan 09 '21 at 13:34

What happens

You do not get the expected html cause it is in an iframe
Try to get the src of the iframe soup.find('iframe')['src'] and request with it again.

Example

import requests
from bs4 import BeautifulSoup
url = "http://www.ingenieur-jobs.de/jobangebote/3075/"
r = requests.get(url)
soup = BeautifulSoup(r.content, 'html.parser')

iframe = requests.get(soup.find('iframe')['src'])

soup = BeautifulSoup(iframe.content, 'html.parser')
soup

Incomplete HTML-response on some sites using Requests & BeautifulSoup or Selenium

1 Answers1