HttpWebResponse.Cookies empty despite Set-Cookie Header (no-redirect)

Question

I'm struggling to figure out what is wrong here. I'm sending login information, I can see the Set-Cookie in the Header with the correct value, but the Cookies collection is not getting filled.

This is HTTPS, the login auto-redirects, but I disabled it with AllowAutoRedirect=false to try to troubleshoot this issue.

In this screenshot, you can easily see the debug information and that the cookie should be getting set. I am setting my httpWebRequest.Cookie to a new CookieCollection.

HttpWebRequest httpRequest;
CookieContainer reqCookies = new CookieContainer();
string url = "https://example.com";
string[] email = user.Split('@');
email[0] = System.Web.HttpUtility.UrlEncode(email[0]);
user = email[0] + "@" + email[1];
pass = System.Web.HttpUtility.UrlEncode(pass);

string postData = "email=" + user + "&password=" + pass;
byte[] byteData = Encoding.UTF8.GetBytes(postData);

httpRequest = (HttpWebRequest)WebRequest.Create(url);
httpRequest.Method = "POST";
httpRequest.Referer = url;
httpRequest.CookieContainer = reqCookies;
httpRequest.UserAgent = "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/535.19 (KHTML, like Gecko) Chrome/18.0.1003.1 Safari/535.19";
httpRequest.Accept = "text/html, application/xhtml+xml, */*";
httpRequest.ContentType = "application/x-www-form-urlencoded";
httpRequest.ContentLength = byteData.Length;
using (Stream postStream = httpRequest.GetRequestStream())
{
    postStream.Write(byteData, 0, byteData.Length);
    postStream.Close();
}

httpRequest.AllowAutoRedirect = false;
HttpWebResponse b = (HttpWebResponse)httpRequest.GetResponse();

Tried the exact same code connecting to http://www.yahoo.com and the cookies are put into my collection... Argh...

Here is the Set-Cookie Header value:

s=541E2101-B768-45C8-B814-34A00525E50F; Domain=example.com; Path=/; Version=1

Answer 1

UPDATE five years later, someone actually mentioned the correct way to do it: setting up the CookieContainer correctly in the first place and letting it handle everything. Please refer to Sam's solution further down.

I've found that issue as well, when reading Cookies in C# that were created by a C# ASP.NET app... ;)

Not sure if it has to do with it, but I found that the two Cookies that are set in my case are written in a single Set-Cookie header, with the cookie payload separated by commas. So I adapted AppDeveloper's solution to deal with this multiple-cookie issue, as well as fixing the name/value thing I mentioned in the comments.

private static void fixCookies(HttpWebRequest request, HttpWebResponse response) 
{
    for (int i = 0; i < response.Headers.Count; i++)
    {
        string name = response.Headers.GetKey(i);
        if (name != "Set-Cookie")
            continue;
        string value = response.Headers.Get(i);
        foreach (var singleCookie in value.Split(','))
        {
            Match match = Regex.Match(singleCookie, "(.+?)=(.+?);");
            if (match.Captures.Count == 0)
                continue;
            response.Cookies.Add(
                new Cookie(
                    match.Groups[1].ToString(), 
                    match.Groups[2].ToString(), 
                    "/", 
                    request.Host.Split(':')[0]));
        }
    }
}

Answer 2

The Cookies property will be null unless the CookieContainer is set on the HttpWebRequest. So the proper way to do this is to set the CookieContainer member before retrieving the response:

var request = (HttpWebRequest)HttpWebRequest.Create(..);
request.CookieContainer = new CookieContainer();

var response = request.GetResponse();
// ..response.Cookies will now contain the cookies sent back by the server.

You don't need to manually parse Set-Cookie.

See the documentation for more information.

Answer 3

It seems like Set-Cookie header sent by the website is malformed (Not in the typical format it should have been).

In such case you need to Parse cookie manually and it it to the CookieContainer.

for (int i = 0; i < b.Headers.Count; i++)
{
    string name = b.Headers.GetKey(i);
    string value = b.Headers.Get(i);
    if (name == "Set-Cookie")
    {
        Match match = Regex.Match(value, "(.+?)=(.+?);");
        if (match.Captures.Count > 0)
        {
            reqCookies.Add(new Cookie(match.Groups[1].Value, match.Groups[2].Value, "/", "example.com"));
        }
    }
}

Answer 4

Use a CookieContainer as in this answer. What tripped these regex approaches up for me was a comma in expires=Tue, ....

Answer 5

Looking on other answers I improved incorrect cookie handling. Unlike those answers this one automatically handles all cookie properties (such as expired, secure, etc.) and works with all range of cookies (even when there are more than 1 incorrect cookie).

It's implemented as extension method and can be used in the following way:

//...
            using (HttpWebResponse response = (HttpWebResponse)request.GetResponse())
            {
                request.FixCookies(response);
//...

FixCookies() extension method:

using System;
using System.Collections.Generic;
using System.Net;

namespace AG.WebHelpers
{
    static public class ExtensionMethods
    {
        static public void FixCookies(this HttpWebRequest request, HttpWebResponse response)
        {
            for (int i = 0; i < response.Headers.Count; i++)
            {
                string name = response.Headers.GetKey(i);
                if (name != "Set-Cookie")
                    continue;
                string value = response.Headers.Get(i);
                var cookieCollection = ParseCookieString(value, () => request.Host.Split(':')[0]);
                response.Cookies.Add(cookieCollection);
            }
        }

        static private CookieCollection ParseCookieString(string cookieString, Func<string> getCookieDomainIfItIsMissingInCookie)
        {
            bool secure = false;
            bool httpOnly = false;

            string domainFromCookie = null;
            string path = null;
            string expiresString = null;

            Dictionary<string, string> cookiesValues = new Dictionary<string, string>();

            var cookieValuePairsStrings = cookieString.Split(';');
            foreach(string cookieValuePairString in cookieValuePairsStrings)
            {
                var pairArr = cookieValuePairString.Split('=');
                int pairArrLength = pairArr.Length;
                for (int i = 0; i < pairArrLength; i++)
                {
                    pairArr[i] = pairArr[i].Trim();
                }
                string propertyName = pairArr[0];
                if (pairArrLength == 1)
                {
                    if (propertyName.Equals("httponly", StringComparison.OrdinalIgnoreCase))
                        httpOnly = true;
                    else if (propertyName.Equals("secure", StringComparison.OrdinalIgnoreCase))
                        secure = true;
                    else
                        throw new InvalidOperationException(string.Format("Unknown cookie property \"{0}\". All cookie is \"{1}\"", propertyName, cookieString));
                    continue;
                }

                string propertyValue = pairArr[1];
                if (propertyName.Equals("expires", StringComparison.OrdinalIgnoreCase))
                    expiresString = propertyValue;
                else if (propertyName.Equals("domain", StringComparison.OrdinalIgnoreCase))
                    domainFromCookie = propertyValue;
                else if (propertyName.Equals("path", StringComparison.OrdinalIgnoreCase))
                    path = propertyValue;
                else
                    cookiesValues.Add(propertyName, propertyValue);
            }

            DateTime expiresDateTime;
            if (expiresString != null)
            {
                expiresDateTime = DateTime.Parse(expiresString);
            }
            else
            {
                expiresDateTime = DateTime.MinValue;
            }
            if (string.IsNullOrEmpty(domainFromCookie))
            {
                domainFromCookie = getCookieDomainIfItIsMissingInCookie();
            }

            CookieCollection cookieCollection = new CookieCollection();
            foreach (var pair in cookiesValues)
            {
                Cookie cookie = new Cookie(pair.Key, pair.Value, path, domainFromCookie);
                cookie.Secure = secure;
                cookie.HttpOnly = httpOnly;
                cookie.Expires = expiresDateTime;

                cookieCollection.Add(cookie);
            }
            return cookieCollection;
        }
    }
}

Answer 6

That could be a bit late, but you can use function SetCookies

var cHeader = responce.Headers.Get("Set-Cookie");
var cookie = new CookieContainer();
cookie.SetCookies(new Uri("[...]"), cHeader);

Answer 7

I know that this question is old, but I came across some code that properly parses a "Set-Cookie" header. It handles cookies separated by commas and extracts the name, expiration, path, value, and domain of each cookie.

This code works better than Microsoft's own cookie parser and this is really what the official cookie parser should be doing. I don't have any clue why Microsoft hasn't fixed this yet since it's a very common issue.

Here is the original code: http://snipplr.com/view/4427/

I'm posting it here in case the link goes down at some point:

public static CookieCollection GetAllCookiesFromHeader(string strHeader, string strHost)
{
    ArrayList al = new ArrayList();
    CookieCollection cc = new CookieCollection();
    if (strHeader != string.Empty)
    {
        al = ConvertCookieHeaderToArrayList(strHeader);
        cc = ConvertCookieArraysToCookieCollection(al, strHost);
    }
    return cc;
}


private static ArrayList ConvertCookieHeaderToArrayList(string strCookHeader)
{
    strCookHeader = strCookHeader.Replace("\r", "");
    strCookHeader = strCookHeader.Replace("\n", "");
    string[] strCookTemp = strCookHeader.Split(',');
    ArrayList al = new ArrayList();
    int i = 0;
    int n = strCookTemp.Length;
    while (i < n)
    {
        if (strCookTemp[i].IndexOf("expires=", StringComparison.OrdinalIgnoreCase) > 0)
        {
            al.Add(strCookTemp[i] + "," + strCookTemp[i + 1]);
            i = i + 1;
        }
        else
        {
            al.Add(strCookTemp[i]);
        }
        i = i + 1;
    }
    return al;
}


private static CookieCollection ConvertCookieArraysToCookieCollection(ArrayList al, string strHost)
{
    CookieCollection cc = new CookieCollection();

    int alcount = al.Count;
    string strEachCook;
    string[] strEachCookParts;
    for (int i = 0; i < alcount; i++)
    {
        strEachCook = al[i].ToString();
        strEachCookParts = strEachCook.Split(';');
        int intEachCookPartsCount = strEachCookParts.Length;
        string strCNameAndCValue = string.Empty;
        string strPNameAndPValue = string.Empty;
        string strDNameAndDValue = string.Empty;
        string[] NameValuePairTemp;
        Cookie cookTemp = new Cookie();

        for (int j = 0; j < intEachCookPartsCount; j++)
        {
            if (j == 0)
            {
                strCNameAndCValue = strEachCookParts[j];
                if (strCNameAndCValue != string.Empty)
                {
                    int firstEqual = strCNameAndCValue.IndexOf("=");
                    string firstName = strCNameAndCValue.Substring(0, firstEqual);
                    string allValue = strCNameAndCValue.Substring(firstEqual + 1, strCNameAndCValue.Length - (firstEqual + 1));
                    cookTemp.Name = firstName;
                    cookTemp.Value = allValue;
                }
                continue;
            }
            if (strEachCookParts[j].IndexOf("path", StringComparison.OrdinalIgnoreCase) >= 0)
            {
                strPNameAndPValue = strEachCookParts[j];
                if (strPNameAndPValue != string.Empty)
                {
                    NameValuePairTemp = strPNameAndPValue.Split('=');
                    if (NameValuePairTemp[1] != string.Empty)
                    {
                        cookTemp.Path = NameValuePairTemp[1];
                    }
                    else
                    {
                        cookTemp.Path = "/";
                    }
                }
                continue;
            }

            if (strEachCookParts[j].IndexOf("domain", StringComparison.OrdinalIgnoreCase) >= 0)
            {
                strPNameAndPValue = strEachCookParts[j];
                if (strPNameAndPValue != string.Empty)
                {
                    NameValuePairTemp = strPNameAndPValue.Split('=');

                    if (NameValuePairTemp[1] != string.Empty)
                    {
                        cookTemp.Domain = NameValuePairTemp[1];
                    }
                    else
                    {
                        cookTemp.Domain = strHost;
                    }
                }
                continue;
            }
        }

        if (cookTemp.Path == string.Empty)
        {
            cookTemp.Path = "/";
        }
        if (cookTemp.Domain == string.Empty)
        {
            cookTemp.Domain = strHost;
        }
        cc.Add(cookTemp);
    }
    return cc;
}