0

I dont know the right term but I think "dynamic website" might do the trick.
By that I mean that as I scroll, resources are being loaded. I searched for solutions and I came across webdrivers, personally I dont like a whole browser being loaded, just for the purpose of scrolling down.
A different approach would be looking at the network tab and scraping the url, that I find there.

https://www.immowelt.de/liste/hamburg/wohnungen/mieten?prima=700&sort=relevanz&cp=1

However the content really gets loaded as I scroll.

the link that gets shown when I open the network tab and scroll down: https://www.immowelt.de/liste/getlistitems

Im new to web development so I dont get how these links that I see in the network tab can be named exactly identical but hold different values.

eyllanesc
  • 235,170
  • 19
  • 170
  • 241
DavidGetter
  • 349
  • 3
  • 12

2 Answers2

1

If you look at the network packet more closely. You will see that it is a POST request and sends form data to that link. Looking at the form data more closely:

query: geoid=108020&etype=1&esr=2&prima=700&sort=relevanz&cp=1
offset: 12
pageSize: 4

You see that it sends an offset. That is what paging the next results.

0

Just looked-up an interesting question (with answers) for a similar problem:
How can I scroll a web page using selenium webdriver in python?

The answers point to the code:

driver.execute_script("window.scrollTo(0, Y)")

There are also infinite scrolling options in the linked Q&A


You might want to establish a parameter of how many entries you want to scrape by setting-up the "scroll" script and printing the output once the scrolling is done.


Hope this helps, cheers!

shongyang low
  • 77
  • 6