Why is SelectSingleNode Causing a System.NullReferenceException for Every Site

Question

I am just trying to figure out webscraping in C#. Right now I am trying to retrieve the user's external IP via HTTP GET API call. I have tried multiple different sites and I have had the same System.NullReferenceException for all of them. Here is my code:

using System;
using HtmlAgilityPack;

namespace Tracer
{
    class IPTracer
    {
        static void Main(string[] args)
        {
            scrape(@"http://api.ipify.org/");
        }

        static void scrape(string url)
        {
            HtmlWeb scraper = new HtmlWeb();
            HtmlDocument html = scraper.Load(url);

            string ip = html.DocumentNode.SelectSingleNode("/html/body/pre").InnerText;
            Console.WriteLine(ip);
        }
    }
}

This might be a very simple mistake as I am rather new to C# but this has been bugging me for days and I have yet to find a solution. And no, I checked and "html" itself is not null.

Answer 1

When I look at the Elements section in the browser's dev tools, I also see that html with an XPath of /html/body/pre. But it looks like the browser dev tools is lying, because when you look at the page source of what http://api.ipify.org/ returns, you'll see that it doesn't return any html at all - it only returns the ip address of the visitor as text.

So given that there is no HTML to parse, you don't need HtmlAgilityPack (and probably can't use) for this web site. You can get the text it returns using HttpClient like this:

using System;
using System.Net.Http;

namespace Tracer
{
    class IPTracer
    {
        static HttpClient client = new HttpClient();


        static void Main(string[] args)
        {
            scrape(@"http://api.ipify.org/");
        }

        static void scrape(string url)
        {
            string ip = "";
            using (HttpResponseMessage response = client.GetAsync(url).Result)
            {
                using (HttpContent content = response.Content)
                {
                    ip =  content.ReadAsStringAsync().Result;
                }
            }

            Console.WriteLine(ip);
        }
    }
}