Web Scraping beautifulsoup not showing node content

Question

I am trying to scrape the following URL : "https://www.licitor.com/ventes-judiciaires-immobilieres/tj-pontoise/mardi-7-septembre-2021.html".

Different items are then available , let say the item number 086276, this gives for example the following URL : https://www.licitor.com/ventes-judiciaires-immobilieres/tj-pontoise/mardi-7-septembre-2021.html#086276

I am particularly interested in the content of the following node:

"<article class="LegalAd"></article>"

I use currently the following code:

URL = "https://www.licitor.com/ventes-judiciaires-immobilieres/tj-pontoise/mardi-7-septembre-2021.html#086276"
page = requests.get(URL)

soup = BeautifulSoup(page.content, "html.parser")

This gives me indeed an output, however, while looking at this output, the content of "LegalAd" is not present. While inspecting the element with Chrome, I can see the html code and navigate inside "LegalAd", but while bringing it into Python, this does not seems to follow...

Does anybody knows why the content of the Nodes gets hidden ? Does anybody would have a fix or an idea ?

@Epsi95 I don't think so -- the element is there but `.text` is empty until JS executes. — ggorlen, Jul 31 '21 at 15:14

score 0 · Accepted Answer · answered Jul 31 '21 at 15:08

The HTML is fetched from another endpoint and injected dynamically into the div using JS. This code just goes directly to the other endpoint where the data is statically present:

import requests
from bs4 import BeautifulSoup

URL = "https://www.licitor.com/annonce/08/62/76/vente-aux-encheres/un-appartement/argenteuil/val-d-oise/086276.html"
response = requests.get(URL)
response.raise_for_status()
soup = BeautifulSoup(response.text, "html.parser")
print(soup.select_one(".LegalAd"))

I found this out by searching for one of the words that appears in the target element in the network tab in developer tools:

The result on top "mardi" appears to be a mobile version.

Web Scraping beautifulsoup not showing node content

1 Answers1