Downloading data with broken header

Question

I just got the following problem:

I need to download data which is behind a login page. When I however make my get request, the server is providing bad data - the content is there, but no content length in header set, its an empty field. I looked it up with Fiddler, and its the same when I try to download the file with browser, but browser completes the download ok, while C# drops with exception when getting the response object from my request.

The header looks like this :

HTTP/1.1 200 OK
Date: Sat, 06 Dec 2014 11:55:06 GMT
Server: Apache
Content-Disposition: attachment; filename=;
Pragma: no-cache
Expires: 0
Content-Length: 
X-Powered-By: PleskLin
Connection: close
Content-Type: application/octet-stream

 Hersteller;"Hersteller Art-Nr";"Lieferant Art-Nr";Ma�stab;Bezeichnung;EAN;"EK (netto)";UVP;USt;Verkaufseinheit;Hinweis;"Letzte Pro...

My code looks like this

    public string ReadPage(string path, string method = "GET"){
        var result = "";
        var request = (HttpWebRequest)WebRequest.Create(path);
        request.Method = method;            
        request.Host = "somehost.de";
        request.CachePolicy = new System.Net.Cache.RequestCachePolicy(System.Net.Cache.RequestCacheLevel.NoCacheNoStore);
        request.Referer = @"http://somehost.de/login.php?redir=list.php%3Ftype%3Dmm";
        request.AllowAutoRedirect = true;
        request.Headers.Add("Cookie", LoginCookie);
        try
        {
        var response = request.GetResponse();           
            using (StreamReader sr = new StreamReader(response.GetResponseStream()))
            {
                result = sr.ReadToEnd();
            }
        }
        catch (Exception ex)
        {
            Console.WriteLine(ex.Message);
            //throw;
        }
        return result;            
    }

The exception appears in the var response = request.GetResponse(); line. Any idea how to fix this problem? I just want it to carry on and let me read out the data.

Forgot the exception - its a WebException with a message The server committed a protocol violation. Section=ResponseHeader Detail='Content-Length' header value is invalid

have a look on http://stackoverflow.com/questions/692342/net-httpwebrequest-getresponse-raises-exception-when-http-status-code-400-ba — metadings, Dec 06 '14 at 13:59
@metadings though that's for a correctly sent error response that is still obeying the rules of HTTP, rather than an incorrectly sent success response. — Jon Hanna, Dec 06 '14 at 19:24
@metadings The OP isn't getting a response, they're getting garbage. — Jon Hanna, Dec 06 '14 at 19:54
I have edited your title. Please see, "[Should questions include “tags” in their titles?](http://meta.stackexchange.com/questions/19190/)", where the consensus is "no, they should not". — John Saunders, Dec 06 '14 at 21:44
@Petr: garbage is garbage. Either the sender needs to fix his garbage, or you need to write your own version of the HTTP protocol which permits such garbage. — John Saunders, Dec 06 '14 at 21:45
@JohnSaunders - Well, thats true, but its what I get from a large distributor's page. I have no fun to sync 10000+ articles to my online shop manually, so have to live with it... — Petr Osipov, Dec 07 '14 at 20:59
No, you don't have to live with it! Does the distributor even _know_ about the problem? Do you think they _want_ to be sending garbage? Let them **know** about it. — John Saunders, Dec 07 '14 at 21:20
I wrote them an email about that a few hours ago, would wonder if and whether they react... — Petr Osipov, Dec 08 '14 at 17:15

score 0 · Answer 1 · edited May 23 '17 at 11:57

I'm really loving the answer of Matthew: .Net HttpWebRequest.GetResponse() raises exception when http status code 400 (bad request) is returned

litte bit changed

public static class WebRequestExtensions
{
    public static WebResponse GetWEResponse(this WebRequest request)
    {
        if (request == null)
        {
            throw new ArgumentNullException("request");
        }

        try
        {
            return request.GetResponse();
        }
        catch (WebException e)
        {
            return e.Response;
        }
    }
}

You can use this now by using request.GetWEResponse

public string ReadPage(string path, string method = "GET") {
    var result = "";
    var request = (HttpWebRequest)WebRequest.Create(path);
    request.Method = method;            
    request.Host = "somehost.de";
    request.CachePolicy = new System.Net.Cache.RequestCachePolicy(System.Net.Cache.RequestCacheLevel.NoCacheNoStore);
    request.Referer = @"http://somehost.de/login.php?redir=list.php%3Ftype%3Dmm";
    request.AllowAutoRedirect = true;
    request.Headers.Add("Cookie", LoginCookie);
    try
    {
        var response = request.GetWEResponse();           
        using (StreamReader sr = new StreamReader(response.GetResponseStream()))
        {
            result = sr.ReadToEnd();
        }
    }
    catch (Exception ex)
    {
        Console.WriteLine(ex.Message);
        //throw;
    }
    return result;            
}

or better

public string ReadPage(string path, string method = "GET")
{
    var request = (HttpWebRequest)WebRequest.Create(path);

    request.Method = method;            
    request.Host = "somehost.de";
    request.CachePolicy = new RequestCachePolicy(RequestCacheLevel.NoCacheNoStore);
    request.Referer = @"http://somehost.de/login.php?redir=list.php%3Ftype%3Dmm";
    request.AllowAutoRedirect = true;
    request.Headers.Add("Cookie", LoginCookie);

    using (var response = (HttpWebResponse)request.GetWEResponse())
    using (var reader = new StreamReader(response.GetResponseStream()))
    {
        return reader.ReadToEnd();
    }
}

Looks interesting, would try it! However, I ran into another problem - my results were when running Fiddler. When fiddler is off, I dont even get that far in login process, need to solve that first... — Petr Osipov, Dec 07 '14 at 21:01

Downloading data with broken header

1 Answers1