Problem scrapping dynamically rendered page

Question

In an ASP.NET MVC application, I am trying to scrap html of a URL like: "https://www.digikala.com/product/dkp-10658066" to get all price and color combinations. The code I am using right now is:

HttpWebRequest request = (HttpWebRequest)WebRequest.Create(producturl);
HttpWebResponse response = (HttpWebResponse)request.GetResponse();
if (response.StatusCode == HttpStatusCode.OK)
{
    Stream receiveStream = response.GetResponseStream();
    StreamReader readStream = null;

    if (String.IsNullOrWhiteSpace(response.CharacterSet))
        readStream = new StreamReader(receiveStream);
    else
        readStream = new StreamReader(receiveStream, Encoding.GetEncoding(response.CharacterSet));

    result = readStream.ReadToEnd();

    response.Close();
    readStream.Close();
}
            
return result;

The problem is, the result is not what I want since contains no price which might be because the page is rendered dynamically. Can anyone help me get the whole HTML of the URL?

The way you are scaping right now, you are making **ONE** request that loads the page, i.e. the HTML code. Interactive elements of this page are **not** triggered by your single request. There is no easy way to get dynamic data like that. You need a headless browser like puppeteer, selenium or phantomjs to *interact* with the dynamic page data. — JaB, Apr 27 '23 at 07:33
https://stackoverflow.com/questions/35365068/c-sharp-download-html-string-after-page-loading-is-finished Hope this helps... — Bhuvaneshwaran, Apr 27 '23 at 08:17

Problem scrapping dynamically rendered page

0 Answers0