XDocument.Load(feedUrl) returns "Data at the root level is invalid. Line 1, position 1."

Question

first of all, this is more of a solution, than a question. It took me a while to resolve, so I thought my resolution might worth sharing.

I was trying to load an RSS feed using XDoument.Load(feedUrl) and was getting the above Exception. I checked the feed itself in my browser and the format looked fine. So after finding some similar situations here:

• Meaning of - <?xml version="1.0" encoding="utf-8"?>
• Why "Data at the root level is invalid. Line 1, position 1." for XML Document?
• LINQ TO XML Parse RSS Feed

and elsewhere: http://www.ipreferjim.com/2014/09/data-at-the-root-level-is-invalid-line-1-position-1/

Answer 1

...I decided it probably wasn't the formatting and that the site in question might be seeing the request as a bot and providing an alternate response.

This was, in fact, the case! I tried fetching the feed with an HttpWebRequest (without setting a useragent) and I received just @ . I tried with a useragent and I got the XML I was after. It was then just a case of using XDocument.Parse()

            XDocument doc;

            try
            {
                doc = XDocument.Load(feedUrl);

            }
            catch (XmlException x)
            {
                string xml = Utilities.WebGetRequest(feedUrl);
                doc = XDocument.Parse(xml);

            }
//carry on working with the doc

...

  public static string WebGetRequest(string url)
  {
      try
      {
          HttpWebRequest request = (HttpWebRequest)HttpWebRequest.Create(url);
          request.UserAgent = "Mozilla/5.0 (compatible; MSIE 9.0; Windows NT 6.1; Trident/5.0)";
          Stream objStream;
          HttpWebResponse response;
          string retVal;
          StreamReader objReader;
          try
          {
              response = request.GetResponse() as HttpWebResponse;
          }
          catch (WebException ex)
          {
              response = ex.Response as HttpWebResponse;
          }
          objStream = response.GetResponseStream();
          objReader = new StreamReader(objStream);
          retVal = objReader.ReadToEnd();

          objReader.Dispose();
          objStream.Dispose();
          response.Dispose();
          return retVal;
      }
      catch (Exception ex)
      {
          //Log("FeedRequest", url, true, ex);   //log it, display it and move on
          return "";
      }
  }