Unable to append 'Vary' header to response

Question

I'm trying to add a Vary: Accept-Encoding header to the response to files that I compress, as advised earlier.

However, for some reason this is not possible - either from the Visual Studio test server, or an IIS server.

I have the following code:

if (url.Contains(".js") || url.Contains(".aspx") || url.Contains(".css"))
{
    app.Response.AppendHeader("Vary", "Accept-Encoding");
    app.Response.AppendHeader("Varye", "Accept-Encoding"); // for testing
    app.Response.AppendHeader("Varye", "Accept-Things");   // for testing
    app.Response.AppendHeader("Vary", "Accept-Stuff");     // for testing
    app.Response.AppendHeader("Var", "Accept-Items");      // for testing

    encodings = encodings.ToLower();

    if (encodings.Contains("gzip") || encodings == "*")
    {
        app.Response.Filter = new GZipStream(baseStream, CompressionMode.Compress);
        app.Response.AppendHeader("Content-Encoding", "gzip");

    }
}

this results in the following response header:

Status=OK - 200
Server=ASP.NET Development Server/10.0.0.0
Date=Fri, 21 Oct 2011 12:24:11 GMT
X-AspNet-Version=4.0.30319
Varye=Accept-Encoding, Accept-Things
Var=Accept-Items
Content-Encoding=gzip
Cache-Control=public
Etag="1CC8F2E9D772300"
Content-Type=text/css
Content-Length=16200
Connection=Close

As you can see, the Vary header is not present. Meaningless headers with similar syntax are present, so there must be something, somewhere, taking out the Vary header before it is sent.

I don't know if it is relevant, but here is where I define my compression module in web.config:

<httpModules>
    <add name="CompressionModule" type="Utility.HttpCompressionModule"/>
</httpModules>

(Where Utility.HttpCompressionModule is the class which the code excerpt I provided above belongs to.)

Why is it that I can't add the Vary header?

EDIT: Eric C's solution has left me with code like this:

if (url.Contains(".js") || url.Contains(".aspx") || url.Contains(".css"))
{
    app.Response.Cache.SetVaryByCustom("Accept-Encoding");

    encodings = encodings.ToLower();

    if (encodings.Contains("gzip") || encodings == "*")
    {
        app.Response.Filter = new GZipStream(baseStream, CompressionMode.Compress);
        app.Response.AppendHeader("Content-Encoding", "gzip");

    }

But, the headers look like this:

Status=OK - 200
Server=ASP.NET Development Server/10.0.0.0
Date=Mon, 24 Oct 2011 09:26:37 GMT
Content-Encoding=gzip
Cache-Control=public
Etag="1CC7A09FDE77300"
Vary=*
Content-Type=application/x-javascript
Content-Length=44447
Connection=Close

(No idea why this is application/x-javascript as its set in the HTML as text/javascript, but this is irrelevant.)

As you can see, I now have a vary header, but it is set as Vary=* rather than Vary=Accept-Encoding, as you would expect from my code in the compression module.

What is going on here? How do I get the Vary header set correctly?

Second Edit: I'm going to paste the source code for the whole class. There isn't much more to it than I have already posted, but it might help to give a grasp of exactly what I'm doing:

public class HttpCompressionModule : IHttpModule
{
    /// <summary>
    /// Initializes a new instance of the <see cref="AjaxHttpCompressionModule"/> class.
    /// </summary>
    public HttpCompressionModule()
    {
    }

    #region IHttpModule Members

    /// <summary>
    /// Disposes of the resources (other than memory) used by the module that implements <see cref="T:System.Web.IHttpModule"/>.
    /// </summary>
    void IHttpModule.Dispose()
    {

    }

    /// <summary>
    /// Initializes a module and prepares it to handle requests.
    /// </summary>
    /// <param name="context">An <see cref="T:System.Web.HttpApplication"/> that provides access to the methods, properties, and events common to all application objects within an ASP.NET application</param>
    void IHttpModule.Init(HttpApplication context)
    {
        context.BeginRequest += (new EventHandler(this.context_BeginRequest));
    }

    #endregion

    /// <summary>
    /// Handles the BeginRequest event of the context control.
    /// </summary>
    /// <param name="sender">The source of the event.</param>
    /// <param name="e">The <see cref="System.EventArgs"/> instance containing the event data.</param>
    void context_BeginRequest(object sender, EventArgs e)
    {            
        HttpApplication app = (HttpApplication)sender;
        string encodings = app.Request.Headers.Get("Accept-Encoding");
        Stream baseStream = app.Response.Filter;


        if (string.IsNullOrEmpty(encodings))
            return;


        string url = app.Request.RawUrl.ToLower();

        if (url.Contains(".js") || url.Contains(".css") || url.Contains("ajax.ashx"))
        {
            app.Response.Cache.SetVaryByCustom("Accept-Encoding");

            encodings = encodings.ToLower();

            if (encodings.Contains("gzip") || encodings == "*")
            {
                app.Response.Filter = new GZipStream(baseStream, CompressionMode.Compress);
                app.Response.AppendHeader("Content-Encoding", "gzip");

            }
            else if (encodings.Contains("deflate"))
            {
                app.Response.Filter = new DeflateStream(baseStream, CompressionMode.Compress);
                app.Response.AppendHeader("Content-Encoding", "deflate");
            }
        }
    }
}

Further, here is the System.Web section of my web.config file:

<system.web>
    <!--<compilation debug="true"></compilation>-->
    <trace enabled="true" traceMode="SortByTime"/>
    <httpRuntime executionTimeout="180"/>
    <globalization culture="en-GB" uiCulture="en-GB"/>
    <!-- custom errors-->
    <customErrors mode="Off">
    </customErrors>
    <!-- Membership -->
    <membership defaultProvider="SqlProvider" userIsOnlineTimeWindow="15">
        <providers>
            <clear/>
            <add name="SqlProvider" type="System.Web.Security.SqlMembershipProvider" connectionStringName="SQLServerAuth" applicationName="mycompany" minRequiredPasswordLength="4" minRequiredNonalphanumericCharacters="0" enablePasswordRetrieval="false" enablePasswordReset="true" requiresQuestionAndAnswer="false" requiresUniqueEmail="true" passwordFormat="Hashed" maxInvalidPasswordAttempts="1024"/>
        </providers>
    </membership>
    <!-- Roles -->
    <roleManager enabled="true" cacheRolesInCookie="true" defaultProvider="SqlProvider">
        <providers>
            <clear/>
            <add connectionStringName="SQLServerAuth" applicationName="mycompany" name="SqlProvider" type="System.Web.Security.SqlRoleProvider"/>
        </providers>
    </roleManager>
    <!-- Authentication -->
    <anonymousIdentification enabled="false"/>
    <authentication mode="Forms">
        <forms name=".AUTH" protection="All" timeout="2" path="/">
        </forms>
    </authentication>
    <httpModules>
        <add name="CompressionModule" type="Utility.HttpCompressionModule"/>
    </httpModules>
</system.web>

There isn't much more to say. There is no other non-standard stuff that we are doing with the site, that I know about. Any ideas?

Answer 1

As of IIS 7.5, the Vary header is overwritten by the gzip IIS filter (gzip.dll), that implements the DyanamicCompressionModule. The filter always sets the header to "Vary: Accept-Encoding", regardless of changes made in ASP.NET code. As of today, the only workaround is to disable compression for dynamic content and then implement it in code. Here's how:

1. Remove the following line from Web.config:

   <add name="CompressionModule" type="Utility.HttpCompressionModule"/>

2. Then go to IIS management console and make sure compression is not enabled for dynamic content.

3. Implement compression manually in Global.asax.cs, method HttpApplication.Application_BeginRequest:

   protected void Application_BeginRequest(object sender, EventArgs e)
   {
       HttpContext context = HttpContext.Current;
       context.Response.Filter 
          = new GZipStream(context.Response.Filter, CompressionMode.Compress);
       context.Response.AppendHeader("Content-Encoding", "gzip");
       context.Response.Cache.VaryByHeaders["Accept-Encoding"] = true;

       // We can now set additional Vary headers...
       context.Response.Cache.VaryByHeaders.UserAgent = true;
       context.Response.Cache.VaryByHeaders["X-Requested-With"] = true;
   }

This is an issue that was recently reported to Microsoft.

Answer 2

Try:

Response.Cache.SetVaryByCustom("Accept-Encoding");
Response.Cache.SetOmitVaryStar(true);

Since it seems in the above that something has convinced the code that * is appropriate here, and you're certain it isn't.

Note that with IE there's a flaw in the caching mechanism (partially fixed, IIRC with IE9 but not before) where it takes such a vary header to indicate total inability to cache (this is valid, but sub-optimal behaviour). For this reason, some people prefer to vary by User-Agent because while Accept-Encoding is the technically correct value to send:

1. IE will cache this correctly.
2. A User-Agent that varies in what Accept-Encoding header it sends is possible, but unlikely in what will be encountered in the field.

Note: You must also make sure that you send different E-Tags with the different versions (g-zipped, deflated and uncompressed) as E-Tags label entities (the bunch of bytes sent, including any Content-Encoding but not including any Transport-Encoding) rather than resources, and not varying the E-Tag can result in much the same sort of SNAFU that you are setting Vary: Accept-Encoding to avoid.

If you can't make it vary the E-Tag, then you'd be better off omitting it and using last-modified alone for conditional GETs.

---

Incidentally, the answer to your wondering about why application/x-javascript is set, is that that is the content-type the server is associating with ".js". The hint in the HTML won't affect that as it isn't something the server sees for this request, though it could be used to over-ride the content-type if something bogus like text/plain was sent by the server (the standards are unclear and user-agent behaviour could vary while remaining within the letter of the standard). Since browsers generally all consider application/javascript application/x-javascript and text/javascript to mean javascript, it matters little here.

Answer 3

You need to manually add the header value to see Vary: Accept-Encoding.

response.Cache.SetOmitVaryStar(true);
response.AddHeader("Vary", "Accept-Encoding");

Answer 4

Try this:

app.Response.Cache.SetVaryByCustom("Accept-Encoding");

Answer 5

I also tried these

Response.Cache.SetOmitVaryStar(true);
Response.AddHeader("Vary", "Accept-Encoding");

, but wouldn't work.. This, however, did work:

Response.Cache.VaryByHeaders["Accept-Encoding"] = true;

Answer 6

As mentioned by others, this is a known issue with the IIS compression module overwriting the Vary header specifically and has been addressed by hotfix.

If you can't ensure the hotfix is installed then another workaround is to append the header using IIS URL Rewrite in your Web.config:

<configuration>
  <system.webServer>
    <rewrite>
      <outboundRules>
        <rule name="Append 'Vary: X-Requested-With' header" patternSyntax="ECMAScript">
          <match serverVariable="RESPONSE_VARY" pattern=".+" />
          <action type="Rewrite" value="{R:0}, X-Requested-With" replace="true" />
        </rule>
        <rule name="Set 'Vary: X-Requested-With' header if no others" patternSyntax="ECMAScript">
          <match serverVariable="RESPONSE_VARY" pattern=".+" negate="true" />
          <action type="Rewrite" value="X-Requested-With" />
        </rule>
      </outboundRules>
    </rewrite>
  </system.webServer>
</configuration>