138

When I try to add a HTTP header key/value pair on a WebRequest object, I get the following exception:

This header must be modified using the appropriate property

I've tried adding new values to the Headers collection by using the Add() method but I still get the same exception.

webRequest.Headers.Add(HttpRequestHeader.Referer, "http://stackoverflow.com");

I can get around this by casting the WebRequest object to a HttpWebRequest and setting the properties such as httpWebReq.Referer ="http://stackoverflow.com", but this only works for a handful of headers that are exposed via properties.

I'd like to know if there's a way to get a finer grained control over modifying headers with a request for a remote resource.

Razor
  • 17,271
  • 25
  • 91
  • 138

13 Answers13

187

If you need the short and technical answer go right to the last section of the answer.

If you want to know better, read it all, and i hope you'll enjoy...


I countered this problem too today, and what i discovered today is that:

  1. the above answers are true, as:

    1.1 it's telling you that the header you are trying to add already exist and you should then modify its value using the appropriate property (the indexer, for instance), instead of trying to add it again.

    1.2 Anytime you're changing the headers of an HttpWebRequest, you need to use the appropriate properties on the object itself, if they exist.

Thanks FOR and Jvenema for the leading guidelines...

  1. But, What i found out, and that was the missing piece in the puzzle is that:

    2.1 The WebHeaderCollection class is generally accessed through WebRequest.Headers or WebResponse.Headers. Some common headers are considered restricted and are either exposed directly by the API (such as Content-Type) or protected by the system and cannot be changed.

The restricted headers are:

  • Accept
  • Connection
  • Content-Length
  • Content-Type
  • Date
  • Expect
  • Host
  • If-Modified-Since
  • Range
  • Referer
  • Transfer-Encoding
  • User-Agent
  • Proxy-Connection

So, next time you are facing this exception and don't know how to solve this, remember that there are some restricted headers, and the solution is to modify their values using the appropriate property explicitly from the WebRequest/HttpWebRequest class.


Edit: (useful, from comments, comment by user Kaido)

Solution is to check if the header exists already or is restricted (WebHeaderCollection.IsRestricted(key)) before calling add

Community
  • 1
  • 1
dubi
  • 2,360
  • 2
  • 22
  • 23
  • 9
    "modify their values using the appropriate property" says it all – CRice Jun 14 '11 at 02:59
  • 83
    This answer is just repeating the exceptions' Message without giving a solution to the problem. – 000 Dec 09 '11 at 02:11
  • 12
    Solution is to check if the header exists already or is restricted (WebHeaderCollection.IsRestricted(key)) before calling add – Kaido Jul 26 '12 at 14:26
  • 8
    @Sam read section 1.1 which solves the issue. that means the property we are trying to add via `Headers.Add()` already exists therefore we should modify it instead. – Junaid Qadir Shekhanzai Jan 23 '13 at 20:51
  • I feel it's important to point out that this *restriction* is a feature of the .NET Framework. I've had some developers assume this error means the HTTP specification somehow restricts the use of these headers, which isn't the case. – Shawn South Sep 25 '14 at 00:05
  • I actually wonder if there is a way to initialize/set the headers at once from a NameValueCollection without using reflections or explicitly using properties. – wodzu Nov 01 '14 at 22:55
  • 5
    "I feel it's important to point out that this restriction is a feature of the .NET Framework" -- I'd rather do not have this kind of feature. – Herberth Amaral Mar 12 '15 at 15:14
  • 3
    This answer says a lot, but doesn't say how you can actually set the header that OP wants to set. – Superbest Apr 06 '15 at 02:31
  • 1
    Chmod's answer below actually gives you want you need. i.e. cast to HttpWebRequest so you can set the property directly. – Lorenz03Tx Dec 01 '15 at 22:48
  • request.Headers.UserAgent.Add(new ProductInfoHeaderValue("my_string")); – pete Jul 23 '22 at 18:39
81

I ran into this problem with a custom web client. I think people may be getting confused because of multiple ways to do this. When using WebRequest.Create() you can cast to an HttpWebRequest and use the property to add or modify a header. When using a WebHeaderCollection you may use the .Add("referer","my_url").

Ex 1

WebClient client = new WebClient();
client.Headers.Add("referer", "http://stackoverflow.com");
client.Headers.Add("user-agent", "Mozilla/5.0");

Ex 2

HttpWebRequest request = (HttpWebRequest)WebRequest.Create(url);
request.Referer = "http://stackoverflow.com";
request.UserAgent = "Mozilla/5.0";
response = (HttpWebResponse)request.GetResponse();
Michał Powaga
  • 22,561
  • 8
  • 51
  • 62
Chmod
  • 1,004
  • 7
  • 4
  • 1
    The Ex 1 solved my problem with this exception. So I changed client.Headers["referer"] = url; to client.Headers.Add("referer", url); and things get working. Thanks. – 000 Dec 09 '11 at 02:10
  • 2
    beware that this answer contains a happy assumption that you are working on desktop .Net runtime and asking for http. The WebRequest.Create can return a variety of different objects depending on what protocol prefix you use. It is related to CustomProtocolHandlers if anyone is interested in them.. And on WP7 or Silverlight the request implementation classes are a little bit different too. Just be careful with this. – quetzalcoatl Apr 26 '12 at 21:28
  • 1
    But I can't modify "Accept" header. How can I modify this? – user Aug 02 '13 at 10:38
  • The first example is still giving me the same error – mrid Aug 18 '19 at 14:11
34

All the previous answers describe the problem without providing a solution. Here is an extension method which solves the problem by allowing you to set any header via its string name.

Usage

HttpWebRequest request = WebRequest.Create(url) as HttpWebRequest;
request.SetRawHeader("content-type", "application/json");

Extension Class

public static class HttpWebRequestExtensions
{
    static string[] RestrictedHeaders = new string[] {
            "Accept",
            "Connection",
            "Content-Length",
            "Content-Type",
            "Date",
            "Expect",
            "Host",
            "If-Modified-Since",
            "Keep-Alive",
            "Proxy-Connection",
            "Range",
            "Referer",
            "Transfer-Encoding",
            "User-Agent"
    };

    static Dictionary<string, PropertyInfo> HeaderProperties = new Dictionary<string, PropertyInfo>(StringComparer.OrdinalIgnoreCase);

    static HttpWebRequestExtensions()
    {
        Type type = typeof(HttpWebRequest);
        foreach (string header in RestrictedHeaders)
        {
            string propertyName = header.Replace("-", "");
            PropertyInfo headerProperty = type.GetProperty(propertyName);
            HeaderProperties[header] = headerProperty;
        }
    }

    public static void SetRawHeader(this HttpWebRequest request, string name, string value)
    {
        if (HeaderProperties.ContainsKey(name))
        {
            PropertyInfo property = HeaderProperties[name];
            if (property.PropertyType == typeof(DateTime))
                property.SetValue(request, DateTime.Parse(value), null);
            else if (property.PropertyType == typeof(bool))
                property.SetValue(request, Boolean.Parse(value), null);
            else if (property.PropertyType == typeof(long))
                property.SetValue(request, Int64.Parse(value), null);
            else
                property.SetValue(request, value, null);
        }
        else
        {
            request.Headers[name] = value;
        }
    }
}

Scenarios

I wrote a wrapper for HttpWebRequest and didn't want to expose all 13 restricted headers as properties in my wrapper. Instead I wanted to use a simple Dictionary<string, string>.

Another example is an HTTP proxy where you need to take headers in a request and forward them to the recipient.

There are a lot of other scenarios where its just not practical or possible to use properties. Forcing the user to set the header via a property is a very inflexible design which is why reflection is needed. The up-side is that the reflection is abstracted away, it's still fast (.001 second in my tests), and as an extension method feels natural.

Notes

Header names are case insensitive per the RFC, http://www.w3.org/Protocols/rfc2616/rfc2616-sec4.html#sec4.2

Joannes Vermorel
  • 8,976
  • 12
  • 64
  • 104
Despertar
  • 21,627
  • 11
  • 81
  • 79
  • i use it for Proxy-Connection, but after it say, yes i contain the key for "Proxy-Connection" it return's null, which lead to null reference exception – Hassan Faghihi Feb 09 '16 at 13:19
  • Thank you for the clever fix. I let the extension set all headers: ```static Dictionary HeaderProperties = new Dictionary(StringComparer.InvariantCultureIgnoreCase); static WebRequestExtensions() { // Get property info for restricted headers. Type type = typeof(HttpWebRequest); foreach (string header in Enum.GetNames(typeof(HttpRequestHeader))) { var property = type.GetProperty(header.ToString()); if (property != null) { HeaderProperties.Add(property.Name, property); } } }``` – Suncat2000 Apr 29 '19 at 16:30
14

I had the same exception when my code tried to set the "Accept" header value like this:

WebRequest request = WebRequest.Create("http://someServer:6405/biprws/logon/long");
request.Headers.Add("Accept", "application/json");

The solution was to change it to this:

HttpWebRequest request = (HttpWebRequest)WebRequest.Create("http://someServer:6405/biprws/logon/long");
request.Accept = "application/json";
Mike Gledhill
  • 27,846
  • 7
  • 149
  • 159
13

Anytime you're changing the headers of an HttpWebRequest, you need to use the appropriate properties on the object itself, if they exist. If you have a plain WebRequest, be sure to cast it to an HttpWebRequest first. Then Referrer in your case can be accessed via ((HttpWebRequest)request).Referrer, so you don't need to modify the header directly - just set the property to the right value. ContentLength, ContentType, UserAgent, etc, all need to be set this way.

IMHO, this is a shortcoming on MS part...setting the headers via Headers.Add() should automatically call the appropriate property behind the scenes, if that's what they want to do.

Michał Powaga
  • 22,561
  • 8
  • 51
  • 62
Jerod Venema
  • 44,124
  • 5
  • 66
  • 109
8

WebRequest being abstract (and since any inheriting class must override the Headers property).. which concrete WebRequest are you using ? In other words, how do you get that WebRequest object to beign with ?

ehr.. mnour answer made me realize that the error message you were getting is actually spot on: it's telling you that the header you are trying to add already exist and you should then modify its value using the appropriate property (the indexer, for instance), instead of trying to add it again. That's probably all you were looking for.

Other classes inheriting from WebRequest might have even better properties wrapping certain headers; See this post for instance.

FOR
  • 4,260
  • 2
  • 25
  • 36
4

Note: this solution will work with WebClientSocket as well as with HttpWebRequest or any other class that uses WebHeaderCollection to work with headers.

If you look at the source code of WebHeaderCollection.cs you will see that Hinfo is used to keep information of all known headers:

private static readonly HeaderInfoTable HInfo = new HeaderInfoTable();

Looking at HeaderInfoTable class, you can notice all the data is stored into hash table

private static Hashtable HeaderHashTable;

Further, in static contructor of HeaderInfoTable, you can see all known headers are added in HeaderInfo array and then copied into hashtable.

Final look at HeaderInfo class shows the names of the fields.

internal class HeaderInfo {

    internal readonly bool IsRequestRestricted;
    internal readonly bool IsResponseRestricted;
    internal readonly HeaderParser Parser;

    //
    // Note that the HeaderName field is not always valid, and should not
    // be used after initialization. In particular, the HeaderInfo returned
    // for an unknown header will not have the correct header name.
    //

    internal readonly string HeaderName;
    internal readonly bool AllowMultiValues;
    ...
    }

So, with all the above, here is a code that uses reflection to find static Hashtable in HeaderInfoTable class and then changes every request-restricted HeaderInfo inside hash table to be unrestricted

        // use reflection to remove IsRequestRestricted from headerInfo hash table
        Assembly a = typeof(HttpWebRequest).Assembly;
        foreach (FieldInfo f in a.GetType("System.Net.HeaderInfoTable").GetFields(BindingFlags.NonPublic | BindingFlags.Static))
        {
            if (f.Name == "HeaderHashTable")
            {
                Hashtable hashTable = f.GetValue(null) as Hashtable;
                foreach (string sKey in hashTable.Keys)
                {

                    object headerInfo = hashTable[sKey];
                    //Console.WriteLine(String.Format("{0}: {1}", sKey, hashTable[sKey]));
                    foreach (FieldInfo g in a.GetType("System.Net.HeaderInfo").GetFields(BindingFlags.NonPublic | BindingFlags.Instance))
                    {

                        if (g.Name == "IsRequestRestricted")
                        {
                            bool b = (bool)g.GetValue(headerInfo);
                            if (b)
                            {
                                g.SetValue(headerInfo, false);
                                Console.WriteLine(sKey + "." + g.Name + " changed to false");
                            }

                        }
                    }

                }
            }
        } 
Sleeper
  • 61
  • 5
  • 2
    Brilliant! This also makes it possible to set those headers for the request used when setting up web sockets and thereby working around this issue: https://github.com/dotnet/corefx/issues/26627 – Øystein Kolsrud Nov 08 '19 at 13:41
  • That should be the case because they all use WebHeaderCollection to manipulate headers. I have tested it on HttpWebRequest only tho. – Sleeper Nov 09 '19 at 19:40
  • i tested with web sockets and it did worked to remove restriction and i can now easily set headers – Gray Programmerz Jul 05 '22 at 07:22
3

The above answers are all fine, but the essence of the issue is that some headers are set one way, and others are set other ways. See above for 'restricted header' lists. FOr these, you just set them as a property. For others, you actually add the header. See here.

    request.ContentType = "application/x-www-form-urlencoded";

    request.Accept = "application/json";

    request.Headers.Add(HttpRequestHeader.Authorization, "Basic " + info.clientId + ":" + info.clientSecret);
Rob
  • 2,363
  • 7
  • 36
  • 54
1

Basically, no. That is an http header, so it is reasonable to cast to HttpWebRequest and set the .Referer (as you indicate in the question):

HttpWebRequest req = ...
req.Referer = "your url";
Marc Gravell
  • 1,026,079
  • 266
  • 2,566
  • 2,900
0

I'm using just:

request.ContentType = "application/json; charset=utf-8"
Stefan Michev
  • 4,795
  • 3
  • 35
  • 30
0

You can just cast the WebRequest to an HttpWebRequest showed below:

var request = (HttpWebRequest)WebRequest.Create(myUri);

and then instead of trying to manipulate the header list, apply it directly in the request property request.Referer:

request.Referer = "yourReferer";

These properties are available in the request object.

Bonomi
  • 2,541
  • 5
  • 39
  • 51
0

I ran into same issue below piece of code worked for me

HttpWebRequest request = (HttpWebRequest)WebRequest.Create(url);

request.Headers["UserAgent"] = "Mozilla/5.0 (compatible; MSIE 9.0; Windows NT 6.1; 
Trident/5.0)"
Vaibhav
  • 31
  • 1
  • 4
0
request.Headers.UserAgent.Add(new ProductInfoHeaderValue("my_string"));
pete
  • 1,878
  • 2
  • 23
  • 43
  • Please read [How do I write a good answer?](https://stackoverflow.com/help/how-to-answer). While this code block may answer the OP's question, this answer would be much more useful if you explain how this code is different from the code in the question, what you've changed, why you've changed it and why that solves the problem without introducing others. – Saeed Zhiany Jul 24 '22 at 03:26