Python webscraping blocked

Question

I want to webscrape german real estate website immobilienscout24.de. I would like to download the HTML of a given URL and then work with the HTML offline. It is not intended for commercial use or publication and I do not intend on spamming the site, it is merely for coding practice. I would like to write a python tool that automatically downloads the HTML of given immobilienscout24.de sites. I have tried to use beautifulsoup for this, however, the parsed HTML doesn't show the content but asks if I am a robot etc., meaning my webscraper got detected and blocked (I can access the site in Firefox just fine). I have set a referer, a delay and a user agent. What else can I do to avoid being detected (i.e. rotating proxies, rotating user agents, random clicks, other webscraping tools that don't get detected...)? I have tried to use my phones IP but got the same result. A GUI webscraping tool is not an option as I need to control it with python. Please give some implementable code if possible. Here is my code so far:

import urllib.request
from bs4 import BeautifulSoup
import requests
import time
import numpy

url = "https://www.immobilienscout24.de/Suche/de/wohnung-mieten?sorting=2#"
req = urllib.request.Request(url, data=None, headers={ 'User-Agent' : 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/86.0.4240.75 Safari/537.36' })
req.add_header('Referer', 'https://www.google.de/search?q=immoscout24)
delays = [3, 2, 4, 6, 7, 10, 11, 17]
time.sleep(numpy.random.choice(delays)) # I want to implement delays like this
page = urllib.request.urlopen(req)
soup = BeautifulSoup(page, 'html.parser')
print(soup.prettify)

username:~/Desktop$ uname -a
Linux username 5.4.0-52-generic #57-Ubuntu SMP Thu Oct 15 10:57:00 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux

Thank you!

Answer 1

I'm the developer of Fredy (https://github.com/orangecoding/fredy). I came across the same issue. After digging into the issue, I found how they check if you're a robot.

First they set a localstorage value.

localstorageAvailable: true

And if it's available, they set a value:

testLocalStorage: 1

If both worked, a cookie is set called reese84=xxx. This is what you want. If you send this cookie with your request, it should work. I've tested it a few times.

Note: This is not yet implemented in Fredy, thus immoscout still doesn't work on the live source as I'm currently rewriting the code.

Answer 2

Try to set Accept-Language HTTP header (this worked for me to get correct response from server):

import requests
from bs4 import BeautifulSoup

url = "https://www.immobilienscout24.de/Suche/de/wohnung-mieten?sorting=2#"

headers = {
    'User-Agent': 'Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:82.0) Gecko/20100101 Firefox/82.0',
    'Accept-Language': 'en-US,en;q=0.5'
}

soup = BeautifulSoup(requests.get(url, headers=headers).content, 'html.parser')

for h5 in soup.select('h5'):
    print(h5.get_text(strip=True, separator=' '))

Prints:

NEU Albertstadt: Praktisch geschnitten und großer Balkon
NEU Sehr geräumige 3-Raum-Wohnung in der Eisenacher Weststadt
NEU Gepflegte 3-Zimmer-Wohnung am Rosenberg in Hofheim a.Taunus
NEU ERSTBEZUG: Wohnung neu renoviert
NEU Freundliche 3,5-Zimmer-Wohnung mit Balkon und EBK in Rheinfelden
NEU Für Singles und Studenten! 2 ZKB mit EBK und Balkon
NEU Schöne 3-Zimmer-Wohnung mit 2 Balkonen im Parkend
NEU Öffentlich geförderte 3-Zimmer-Neubau-Wohnung für die kleine Familie in Iserbrook!
NEU Komfortable, neuwertige Erdgeschosswohnung in gefragter Lage am Wall
NEU Möbliertes, freundliches Appartem. TOP LAGE, S-Balkon, EBK, ruhig, Schwabing Nord/, Milbertshofen
NEU Extravagant & frisch saniert! 2,5-Zimmer DG-Wohnung in Duisburg-Neumühl
NEU wunderschöne 3 Zimmer Dachgeschosswohnung mit Einbauküche. 2er WG-tauglich.
NEU Erstbezug nach Sanierung: Helle 3-Zimmer-Wohnung mit Balkon in Monheim am Rhein
NEU Morgen schon im neuen Zuhause mit der ganzen Familie! 3,5 Raum zur Miete in DUI-Overbruch
NEU Erstbezug: ansprechende 2-Zimmer-EG-Wohnung in Bad Düben
NEU CALENBERGER NEUSTADT | 3-Zimmer-Wohnung mit großem Süd-Balkon
NEU Wohnen und Arbeiten in Bestlage von HH-Lokstedt !
NEU Erstbezug: Wohlfühlwohnen in modernem Dachgeschoss nach kompletter Sanierung!
NEU CASACONCEPT Stilaltbau-Wohnung München-Bogenhausen nahe Prinzregentenplatz
NEU schöne Wohnung mit Balkon und Laminatboden

Answer 3

Coming back to this question after a while...

For your information, I brought back support for Immoscout in Fredy. Have a look here. https://github.com/orangecoding/fredy#immoscout

Answer 4

Maybe have a go with requests, the code below seems to work fine for me:

import requests
from bs4 import BeautifulSoup

r = requests.get('https://www.immobilienscout24.de/')

soup = BeautifulSoup(r.text, 'html.parser')
print(soup.prettify)

Another approach is to use selenium; it's powerful but maybe a bit more complicated.

Edit:

A possible solution using Selenium (it seems to work for me for the link you provided in the comment):

from selenium import webdriver
from bs4 import BeautifulSoup

driver = webdriver.Chrome('path/to/chromedriver') # it can also work with Firefox, Safari, etc
driver.get('some_url')
soup = BeautifulSoup(driver.page_source, 'html.parser')

If you haven't used selenium before, have a look here first on how to get started.