Empty response data using HttpWebResponse with Mono on Ubuntu

Question

I'm trying to run a .NET program on Ubuntu using mono.

Program connects to remote server API and gets XML string as response. Here's a simplified version of function responsible for this task.

 using System.Net;

 static string GetProducts(string clientID, string apiKey, string url)
    {
        HttpWebRequest req = WebRequest.Create(url) as HttpWebRequest;
        req.Credentials = new NetworkCredential(clientID, apiKey);
        req.ContentType = "application/x-www-form-urlencoded";

        string result = string.Empty;

        using (HttpWebResponse resp = req.GetResponse() as HttpWebResponse)
        {
            StreamReader reader = new StreamReader(resp.GetResponseStream());
            result = reader.ReadToEnd();
        }

        return result;
    }

It works perfectly well on my Windows 8.1 machine. But my goal is to run it on Ubuntu VPS. I'm using mono to achieve that. Program runs, but stops with exception when it tries to parse downloaded XML string.

[ERROR] FATAL UNHANDLED EXCEPTION: System.Xml.XmlException: Document element did not appear.  Line 1, position 1.

With a little probing I found out that the program isn't actually getting response in XML but produces an empty string instead. Which is odd as no connection errors are being thrown.

I've had some previous experience with mono and Ubuntu, but never ran into problem like this before.

Could it be something to do with Ubuntu server or mono? Or is in the code itself?

Any thoughts on this?

You could try something like NetTool to analyze the traffic ubuntu and then compare that to the traffic you see on Fiddler in windows. Maybe there's some default headers/settings you're relying on in windows that might need to be explicitly set in linux. — cgotberg, Apr 07 '14 at 17:47
@cgotberg After some testing I found out that headers sent are a bit different in windows and ubuntu. on Windows `Content-Type: application/x-www-form-urlencoded Authorization: Basic MjY2MjQyOmFmNDhlZjUwZDFiM2Y5MWRiZmE5ZTAwYTNjM2ZhNjg3` on Ubuntu `Content-Type: application/x-www-form-urlencoded Connection: keep-alive` Looks like Ubuntu is missing Authorization line. But I don't set it myself anywhere on Windows. Can this be of any importance? — Aceonace, Apr 07 '14 at 21:19
It is likely the problem. This line sets the authorization, and it appears that's what's not working in Ubuntu. req.Credentials = new NetworkCredential(clientID, apiKey); Unfortunately I don't have have any experience in why that would be the case. Might be able to try something like this solution. http://stackoverflow.com/a/19851527/1181408 — cgotberg, Apr 08 '14 at 19:39
@cgotberg Thanks for your reply. Indeed req.Credentials doesn't seem to work with Ubuntu. Following your posted solution I've added Authorization header manually. Now headers are more or less identical in each system. Unfortunately the problem still persists. I'm not getting any response data on Ubuntu. Can it be server issue instead of the program? — Aceonace, Apr 09 '14 at 12:17
It could be a lot of things although you would think that identical requests to a web service server would result in the same responses no matter where they came from. Are you getting any response when you make the call from ubuntu? If you get absolutely no acknowledgement back it could be something like firewall (i.e. the server refuses to send a response to the ubuntu machine ip). If you are getting a response but it's empty than that makes the problem tougher to solve. — cgotberg, Apr 09 '14 at 13:43
@cgotberg At first I thought I was getting empty string. Though after further testing I'm now convinced there's just no response from web service. This may be firewall issue after all. — Aceonace, Apr 09 '14 at 16:04

score 1 · Accepted Answer · edited May 23 '17 at 10:25

At last I found where the problem was and a somewhat of a solution to it. First of all I wrote a simple code in Perl to see where my problem was located: in mono or linux server itself.

#!/usr/bin/perl 
use LWP::UserAgent 6;
my $ClientID = "id";
my $APIKey = "key";
my $ua = LWP::UserAgent->new(ssl_opts => { verify_hostname => 0 });
my $req = HTTP::Request->new(GET => 'https://server.url');
$req->authorization_basic($ClientID, $APIKey);
my $res = $ua->request( $req );
print "Status: " . $res->status_line . "\n";
print "Contents: ". $res->content . "\n";

Perl code worked, giving me OK on request status and printing out contents. But to get it working I had to disable server security check LWP::UserAgent->new(ssl_opts => { verify_hostname => 0 }), otherwise it wouldn't work. That got me thinking. Maybe that's why I wasn't getting any response data in my C# code. Because of failed security check. After further testing I got my confirmation that it was indeed my case. I couldn't find why ssl security check failed, so I decided to ignore the check completely as I did in my Perl code. I did a little research and found this thread How to ignore SSL Certificate check. I just added this piece of code to my function and it got it working.

System.Net.ServicePointManager.ServerCertificateValidationCallback +=
delegate(object sender, System.Security.Cryptography.X509Certificates.X509Certificate certificate,
                    System.Security.Cryptography.X509Certificates.X509Chain chain,
                    System.Net.Security.SslPolicyErrors sslPolicyErrors)
{
    return true; // **** Always accept
};

I still don't know why certificate check fails on linux. But at least, by ignoring the check, it works on both Windows and Linux. Here's the final code:

using System.Net;

static string GetProducts(string clientID, string apiKey, string url)
{
    HttpWebRequest req = WebRequest.Create(url) as HttpWebRequest;
    req.Credentials = new NetworkCredential(clientID, apiKey);
    req.ContentType = "application/x-www-form-urlencoded";

    System.Net.ServicePointManager.ServerCertificateValidationCallback +=
    delegate(object sender, System.Security.Cryptography.X509Certificates.X509Certificate certificate,
                    System.Security.Cryptography.X509Certificates.X509Chain chain,
                    System.Net.Security.SslPolicyErrors sslPolicyErrors)
    {
        return true;
    };

    string result = string.Empty;

    using (HttpWebResponse resp = req.GetResponse() as HttpWebResponse)
    {
        StreamReader reader = new StreamReader(resp.GetResponseStream());
        result = reader.ReadToEnd();
    }

    return result;
}

Empty response data using HttpWebResponse with Mono on Ubuntu

1 Answers1