Why does `node` return null?

Question

I am trying to scrape a website using HTML Agility Pack, but have run into an error. The node variable always throws a NullReferenceException when debugging. Why?

 static void Main ( string [] args )
    {
        var html = @"https://www.amazon.com/s?k=";
        Console.WriteLine("Enter a product:  ");
        string item = Console.ReadLine();
        string closing = @"&ref=nb_sb_noss_2";

        item = item.Replace(' ', '+');

        var uri = new Uri(html + item + closing);

        HtmlWeb web = new HtmlWeb();

        HtmlDocument htmlDoc = web.Load(uri);
        var node = htmlDoc.DocumentNode.SelectNodes($"//div[@class='sg-col-inner']");
        double price = 0.0;
        string spWhole, spDecimal, name, bestName = "";
        int myI = 0;

        foreach (HtmlNode product in node)
        {
            Console.Write(myI + "... ");
            name = product.SelectSingleNode("//span[@class='a-size-medium a-color-base a-text-normal']").InnerText;
            spWhole = product.SelectSingleNode("//span@[class='a-price-whole']").InnerText;
            spDecimal = product.SelectSingleNode("//span[@class='a-price-fraction']").InnerText;
            double nPrice = Convert.ToDouble(spWhole);
            nPrice += Convert.ToDouble(spDecimal)/100;

            if (nPrice > price)
            {
                price = nPrice;
                bestName = name;
            }

            using (StreamWriter sw = new StreamWriter("Prices.txt"))
            {
                sw.WriteLine(name + ":  " + nPrice);
            }
            myI++;
        }

        Console.WriteLine(bestName + ":  " + price);


        Console.Read();
    }

obviously your xPath is invalid and no node is found. However I don´t have any clue on your DOM and thus what would be the correct path. — MakePeaceGreatAgain, Feb 22 '20 at 20:12
`$"//div[@class='sg-col-inner']"` seems to be incorrect. And NO, I don´t know the correct one, as mentioned above. Maybe there s no div of that class. — MakePeaceGreatAgain, Feb 22 '20 at 20:14
This is because the scripts are not executed by HAP. If you are using chrome, which renders the page and executes the scripts, you will see different html than what HAP gives you. — Jawad, Feb 22 '20 at 20:52
I dont believe there is any easy way but I would recommend searching for tools that might help you render the page with javascript before you scrape it — Jawad, Feb 22 '20 at 22:18

score 1 · Answer 1 · answered Feb 22 '20 at 20:18

It looks like no results were found when you are querying using that XPath expression, a null response is a completely valid response when calling DocumentNode.SelectNodes

An HtmlAgilityPack.HtmlNodeCollection containing a collection of nodes matching the HtmlAgilityPack.HtmlNode.XPath query, or null if no node matched the XPath expression.

When writing Xpath you might find it useful to use a plugin to test your xpath on the browser. I've used this Xpath Finder plugin before

I used the plugin but it still returns the error – kaptcha Feb 22 '20 at 21:04 — kaptcha, Feb 22 '20 at 21:04

Why does `node` return null?

1 Answers1