How to get verbose info of the HTTP request from HttpClient like those from libcurl?

Question

I am writing a WinForms program (C#) using System.Net.Http.HttpClient to download webpages.

I'd like to get verbose info of the HTTP(S) requests like the following from libcurl:

*    Trying xxx.xxx.xxx.xxx...
*  TCP_NODELAY set
*  Connected to xxx.xxx.xxx.xxx (xxx.xxx.xxx.xxx) port xxxx (#3)
*  allocate connect buffer!
*  Establish HTTP proxy tunnel to www.amazon.com:443
-> CONNECT www.amazon.com:443 HTTP/1.1
-> Host: www.amazon.com:443
-> User-Agent: libcurl/7.64.1 r-curl/4.3.2 httr/1.4.2
-> Proxy-Connection: Keep-Alive
-> 
<- HTTP/1.1 200 Connection established
<- 
*  Proxy replied 200 to CONNECT request
*  CONNECT phase completed!
*  CONNECT phase completed!
*  CONNECT phase completed!
-> GET /xxxxxxx HTTP/1.1
-> Host: www.amazon.com
-> User-Agent: libcurl/7.64.1 r-curl/4.3.2 httr/1.4.2
-> Accept-Encoding: deflate, gzip
-> Cookie: session-id-time=xxxxxx; i18n-prefs=USD; session-id=xxxx; sp-cdn="L5Z9:SG"
-> Accept: application/json, text/xml, application/xml, */*
-> 
<- HTTP/1.1 200 OK
<- Server: Server
<- Content-Type: text/html;charset=UTF-8
<- x-amz-rid: xxxx
<- X-Content-Type-Options: nosniff
<- Expires: -1
<- X-XSS-Protection: 1;
<- Content-Language: en-US
<- Pragma: no-cache
<- Accept-CH-Lifetime: 86400
<- Cache-Control: no-cache
<- Content-Encoding: gzip
<- Accept-CH: ect,rtt,downlink
<- Strict-Transport-Security: max-age=47474747; includeSubDomains; preload
<- Vary: Content-Type,X-Amazon-Wtm-Tag-SP-Search-Secured-Port-Enabled,Accept-Encoding,X-Amzn-CDN-Cache,X-Amzn-AX-Treatment,User-Agent
<- X-Frame-Options: SAMEORIGIN
<- Permissions-Policy: interest-cohort=()
<- Date: Tue, 04 Jan 2022 05:06:02 GMT
<- Transfer-Encoding:  chunked
<- Connection: keep-alive
<- Connection: Transfer-Encoding
*  Added cookie ubid-main="xxxx" for domain amazon.com, path /, expire 1672808762
<- Set-Cookie: ubid-main=xxx; Domain=.amazon.com; Expires=Wed, 04-Jan-2023 05:06:02 GMT; Path=/; Secure
<- 
*  Connection #3 to host xxx.xxx.xxx.xxx left intact

How do I get this kind of message in real time? I am not looking for capturing exceptions.

I understand there are answers using the app.config to create logs, but this kind of solution is not useful here because the log file is locked while the program is running. So I can't really read the log file of the program and show the content in the same program.

Answer 1

From the moment you can't use the logs, a good approach in this case, would be to create a function which is going to print all the data that you need just like libcurl. You may not get the info in real time just like curl or the logging but you'll have the data available to be used in your code without reading another file.

There is also CurlSharp but it doesn't receive a commit since 2017, although it contains many features used in the official libcurl. Beside the commits, you would also need to update the .NET version and the code.

Let's say that you want to do a Get Request, you would do something like this:

    HttpClient client = new HttpClient();
    HttpResponseMessage response = await client.GetAsync("http://www.contoso.com/");
    response.EnsureSuccessStatusCode();

With the HttpResponseMessage you can print many properties related to the request, in particular with the Headers property, you can use the HttpResponseHeaders class to print other properties such as the Connection, the Pragma or the Server. Here's an example:

try
{
    HttpClient client = new HttpClient();
    HttpResponseMessage response = await client.GetAsync("http://www.contoso.com/");
    response.EnsureSuccessStatusCode();

    //if you need it
    //string responseBody = await response.Content.ReadAsStringAsync();

    PrintHttpGetResponse(response);
}
catch (HttpRequestException e)
{
    Console.WriteLine("\nException Caught!");
    Console.WriteLine("Message :{0} ", e.Message);
}

void PrintHttpGetResponse(HttpResponseMessage response) {
    Console.WriteLine($"Content Type ---> {response.Content.Headers.ContentType}");
    Console.WriteLine($"Status Code ---> {response.StatusCode}");
    Console.WriteLine($"Headers ---> {response.Headers}");
    Console.WriteLine($"Connection ---> {response.Headers.Connection}");
    Console.WriteLine($"IsSuccessStatusCode ---> {response.IsSuccessStatusCode}");
}

You could also use some kind of Model with all the properties that you need, to save the response content and share it with other layers or classes easily in your code. For example:

using System.Net;
using System.Net.Http.Headers;

public class ResponseContent {
    MediaTypeHeaderValue ContentType { get; set; }
    HttpStatusCode StatusCode { get; set; }
    HttpResponseHeaders Headers { get; set; }
    HttpHeaderValueCollection<string> Connection { get; set; }
    bool IsSuccessStatusCode { get; set; }
}