HttpWebRequest is not sending all cookies

Question

I'm trying to have my application perform a login action on an external website. I use the following code:

Dim enc As Encoding = Encoding.UTF8
    Dim Data As Byte() = Nothing
    Dim req As HttpWebRequest

    req = CType(Net.WebRequest.Create(URL), Net.HttpWebRequest)
    req.Method = method
    req.CookieContainer = CookieJar

    req.AllowAutoRedirect = False
    If method = "POST" Then
        req.ContentType = "application/x-www-form-urlencoded"
        Data = enc.GetBytes(PostData)
        If Data.Length > 0 Then
            req.ContentLength = Data.Length
            Dim newStream As Stream = req.GetRequestStream()
            newStream.Write(Data, 0, Data.Length)
            newStream.Flush()
            newStream.Close()
        End If
    End If

    Dim Response As Net.HttpWebResponse = CType(req.GetResponse(), Net.HttpWebResponse)

    Dim ResponseStream As IO.StreamReader = New IO.StreamReader(Response.GetResponseStream(), enc)
    Dim Html As String = ResponseStream.ReadToEnd()

    Response.Close()
    ResponseStream.Close()

    Return Html

What works:

• The responses have all the proper "Set-Cookie" headers
• The container saves all the right cookies (5 in total)

What doesn't work:

• All cookies are correctly being retrieved by the container. But not all cookies are sent along whith the next request. 4 cookies are set correctly but the most important one is not sent.

The cookie that is not send is this one:

Set-Cookie: mpSecurity="ODc2NzM2ODoxMzUODViNTg5OWM1NTNlOWMwYmMxYjUxNWZjYzJjOGQyZGU4MTc2M2M=";Version=1;Path=/;Domain=.xxxxx.nl;Discard

The only difference between this cookie and the cookies that are correctly sent is that this one has "Version=1" and "Discard" in it...

Does anybody have any idea why all retrieved cookies are sent except for the one above?

Any help would be appreciated!

Answer 1

This is a common known bug in CookieContainer : Link Here for .Net version below 4.0

Notice the Domain of Set-Cookie Header:

Cookie # 1 -> Set-Cookie: mpSecurity="ODc2NzM2ODoxMzUODViNTg5OWM1NTNlOWMwYmMxYjUxNWZjYzJjOGQyZGU4MTc2M2M=";Version=1;Path=/;Domain=marktplaats.nl;Discard
Cookie # 2 -> Set-Cookie: mpSecurity="ODc2NzM2ODoxMzUODViNTg5OWM1NTNlOWMwYmMxYjUxNWZjYzJjOGQyZGU4MTc2M2M=";Version=1;Path=/;Domain=.marktplaats.nl;Discard

Cookie #1 is sent when the URL format is like http://marktplaats.nl/...
Cookie #2 is sent when the URL format is like http://www.marktplaats.nl/...

Hence the problem

---

Here the solution # 1: (better and easy one)

    class DomainComparer : StringComparer
    {
        public override int Compare(string x, string y)
        {
            if (x == null || y == null)
            {
                return StringComparer.OrdinalIgnoreCase.Compare(x, y);
            }
            if (x.StartsWith("www.", StringComparison.OrdinalIgnoreCase))
            {
                x = x.Substring(4);
            }
            if (y.StartsWith("www.", StringComparison.OrdinalIgnoreCase))
            {
                y = y.Substring(4);
            }
            return StringComparer.OrdinalIgnoreCase.Compare(x, y);
        }

        public override bool Equals(string x, string y)
        {
            return Compare(x, y) == 0;
        }

        public override int GetHashCode(string obj)
        {
            if (obj.StartsWith("www.", StringComparison.OrdinalIgnoreCase))
            {
                obj = obj.Substring(4);
            }
            return StringComparer.OrdinalIgnoreCase.GetHashCode(obj);
        }
    }

    /// <summary>
    /// this is a hackfix for microsoft bug, where cookies are not shared between www.domain.com and domain.com
    /// </summary>
    /// <param name="cc"></param>
    static void ImproveCookieContainer(ref CookieContainer cc)
    {
        Hashtable table = (Hashtable)cc.GetType().InvokeMember(
            "m_domainTable",
            System.Reflection.BindingFlags.NonPublic | System.Reflection.BindingFlags.GetField | System.Reflection.BindingFlags.Instance,
            null, cc, new object[] { });
        var comparerPreperty = table.GetType().GetField("_keycomparer", 
            System.Reflection.BindingFlags.NonPublic | System.Reflection.BindingFlags.GetField | System.Reflection.BindingFlags.Instance);
        if (comparerPreperty != null)
        {
            comparerPreperty.SetValue(table, new DomainComparer());
        }
    }

Implementation of Solution # 1, whenever you create a instance of CookieContainer just call the method once

void main()
{
    CookieContainer cookieJar = new CookieContainer();
    ImproveCookieContainer(ref cookieJar);
    // then use it with the WebRequest object
}

---

Here the solution # 2:

1. Don't use .Add(Cookie), Use only .Add(Uri, Cookie) method.

2. Call BugFix_CookieDomain each time you add a cookie to the container or before you use .GetCookie or before system use the container.

   private void BugFix_CookieDomain(CookieContainer cookieContainer)
   {
       System.Type _ContainerType = typeof(CookieContainer);
       Hashtable table = (Hashtable)_ContainerType.InvokeMember("m_domainTable",
                                  System.Reflection.BindingFlags.NonPublic |
                                  System.Reflection.BindingFlags.GetField |
                                  System.Reflection.BindingFlags.Instance,
                                  null,
                                  cookieContainer,
                                  new object[] { });
       ArrayList keys = new ArrayList(table.Keys);
       foreach (string keyObj in keys)
       {
           string key = (keyObj as string);
           if (key[0] == '.')
           {
               string newKey = key.Remove(0, 1);
               table[newKey] = table[keyObj];
           }
       }
   }
   

All Credit for the solution to CallMeLaNN

Answer 2

Decided to post another workaround for this/a similar case, in case someone finds it useful.

I've had the same problem, received three cookies, only two were showing up as being sent from the next web request. I did some debugging and found that the error for me wasn't the domain itself as the previous answer suggests - in my case the problem was that the third cookie that was returned and was not reused in the subsequent requests was set on an HTTPS redirect, and it had the "Secure" property set to true. So all I had to do was to change the next request url from http://.... to https://...., then all cookies were sent over, and everything worked.

Hope it helps someone!