1

the list Li should have the 116 elements but instead it has 0 elements

import requests
from bs4 import BeautifulSoup
url='https://www.profile.nl/zoekresultaten/banden?size_width=205&size_height=60&size_inch=16'
profile= requests.get(url)
if profile.ok:

    soup = BeautifulSoup(profile.text,'lxml')

    Li= (soup.body).findAll("div", attrs={"class": "row xs-ptb-1"})
    print(len(Li))
wissem Ayat
  • 11
  • 1
  • The class tag you are looking for does not exist on that page returned by BeautifulSoup. Search through `soup` and you will see. Change the attributes you are using to search for the content you want extract. – Rich Lysakowski PhD Jul 11 '21 at 05:46

1 Answers1

4

It is being dynamically pulled from another endpoint you can see in network tab. With a browser this additional request is used to update the page. With requests, this does not happen; so you make the request direct to that other endpoint to get the info.

import requests

r = requests.get('https://www.profile.nl/tyres_search.json?locale=nl&size_width=205&size_height=60&size_inch=16.0').json()
print(r['products'])
QHarr
  • 83,427
  • 12
  • 54
  • 101