Save/Load a "Browser"-session of HttpClient in c#

Question

I have a Webcrawler application that uses HttpClient in order to do some requests.

When i start the application, a new HttpClient is initialized similar to this:

public static void ResetHttpClient()
{
    Client = new HttpClient();
}

After an application restart, the application has to log in freshly which causes trouble due to Limitations to n logins per hour.

Because of that I copied a function from another part of my Apllication that Serializes a List> and saves this list to a file.

I adapted the function in order to test it with HttpClient. My intention was to save the HttpClient and later, when the Program starts/restarts, to load the HttpClient and continue the "Session". Code for the Save function (i know its very ugly with the List just for one entry):

// SAVE
(HttpClient client, DateTime clientCreationDate) httpClientSaveData = 
    new ValueTuple<HttpClient, DateTime>(Client, ClientCreatedDate);
FileStream stream = new FileStream(@"HttpClient.dat", FileMode.Create);
BinaryFormatter formatter = new BinaryFormatter();
formatter.Serialize(stream, httpClientSaveData);
stream.Close();

With this code above I am receiving the following errormessage:

System.Runtime.Serialization.SerializationException: 'Type 'System.Net.Http.HttpClient' in Assembly 'System.Net.Http, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a' is not marked as serializable.'

Now my question is: Is it even possible to save/load a "Session", eg by saving the cookies? What could be a viable approach to this?

*Cookies* (Via an HTTP Header) is the mechanism by which an HTTP Request is reconciled to a session by the server, these are read/writable with HttpClient, E.g. https://stackoverflow.com/questions/13318102/struggling-trying-to-get-cookie-out-of-response-with-httpclient-in-net-4-5 — Alex K., Aug 29 '18 at 14:03
I would imagine the login process probably issues a cookie in the expectation that the browser would store it for use in future requests. This is how the majority of browser-based login systems work. You might be able to get hold of that and then re-use it, until it expires. I'm afraid off the top of my head I don't know how to do that, but maybe if you search about using cookies with C# HttpClient you might start yourself down the right track. — ADyson, Aug 29 '18 at 14:03

ste-fu · Answer 1 · 2018-08-29T14:51:47.230

The error is occurring because you are trying to Serialize an object that does not have the [Serializable] attribute. Because it is a framework class you cannot add it either.

The HttpClient object is meant to be reused, it is just a question of how in your application. Generally you would wrap the framework HttpClient in a class that does the specific action you require, and use it as a Singleton.

It looks as if that is more or less what you are doing, so the issue is more that you application is starting and stopping frequently, which is causing the object to be recreated.

If you want to run something periodically while maintaining state then the best tool for the job is a Windows Service

If you are firing off your request based on receiving another request in you own web app, then you should look to use Dependency Injection, with a Singleton scope. This will keep the object alive as long as the App Pool is enabled. You can increase the Idle time in IIS if necessary. I think it's 15 mins by default, which would guarantee no more than 4 logins per hour.

EDIT:

As the main issue appears to be be during development and testing, you should be able to cheat a bit. Using a proxy like Fiddler you can record the steps taken to log in manually. What you need to pay attention to are the Http Headers. Most authentication / session type data will either be in the Cookie header or the Authorization header. There may also be headers like X-Session-ID or similar. You should be able to manually copy the headers after a successful login and then paste them into your code. A client like Postman can be very useful here, for determining exactly which headers are required.

You code could then look like this:

public static void ResetHttpClient()
{
    Client = new HttpClient();

    if (isDevelopment) // however you determine it
    {
         Client.DefaultRequestHeaders
               .TryAddWithoutValidation("Authorization", "Bearer hhxcshRT43bhJHjh53hkjhbhvghf54g5hbhkghfdUGUKGYFDJihu5");
         Client.DefaultRequestHeaders
               .TryAddWithoutValidation("Cookie", "SessionID=36dab-3545-adef");
    }
}

Basically the (desktop) program itself is not restarting all that often, it's ment to run as stable and long as possible. The issue is that the Application requires much development and testing. When debugging new features or Bugfixes, I have to relaunch the application, of course. Since the remote sever is not in my hands, I cannot change the login limitation and testing my code once in an hour is unfortunately not feasible for me. — julian bechtold, Aug 29 '18 at 14:30

score 0 · Accepted Answer · answered Sep 12 '18 at 12:38

The Solution that worked for me was to save the CookieContainer to a file and load it. I created the following two functions for this purpose:

/// <summary>
/// this function loads the CookieContainer that contains the Session Cookies
/// </summary>
public static void LoadHTTPClient()
{
    CookieJar = new CookieContainer();
    if (File.Exists(@"ClientSessionCookies.dat"))
    {
        // LOAD
        FileStream inStr = new FileStream(@"ClientSessionCookies.dat", FileMode.Open);
        BinaryFormatter bf = new BinaryFormatter();
        CookieJar = bf.Deserialize(inStr) as CookieContainer;
        inStr.Close();
    }
    var handler = new HttpClientHandler
    {
        CookieContainer = CookieJar,
        UseCookies = true,
    };
    Client = new HttpClient(handler);
}

/// <summary>
/// this function saves the sessioncookies to file
/// </summary>
/// <param name="CookieJar"></param>
public static void SaveHTTPClient(CookieContainer CookieJar)
{
    // SAVE client Cookies
    FileStream stream = new FileStream(@"ClientSessionCookies.dat", FileMode.Create);
    BinaryFormatter formatter = new BinaryFormatter();
    formatter.Serialize(stream, CookieJar);
    stream.Close();
}

thx @AlexK and @ADyson for the Hint

I would imagine the login process probably issues a cookie in the expectation that the browser would store it for use in future requests. This is how the majority of browser-based login systems work. You might be able to get hold of that and then re-use it, until it expires. I'm afraid off the top of my head I don't know how to do that, but maybe if you search about using cookies with C# HttpClient you might start yourself down the right track. – ADyson

-

Cookies (Via an HTTP Header) is the mechanism by which an HTTP Request is reconciled to a session by the server, these are read/writable with HttpClient, E.g. stackoverflow.com/questions/13318102/… – Alex K.

Save/Load a "Browser"-session of HttpClient in c#

2 Answers2