Getting Varnish To Work on Magento

Question

First please forgive me for total lack of understanding of Varnish. This is my first go at doing anything with Varnish.

I am following the example at: http://www.kalenyuk.com.ua/magento-performance-optimization-with-varnish-cache-47.html

However when I install and run this, Varnish does not seem to cache. I do get the X-Varnish header with a single number and a Via header that has a value of 1.1 varnish

I have been told (by my ISP) it is because of the following cookie that Magento sets:

Set-Cookie: frontend=6t2d2q73rv9s1kddu8ehh8hvl6; expires=Thu, 17-Feb-2011 14:29:19 GMT; path=/; domain=XX.X.XX.XX; httponly

They said that I either have to change Magento to handle this or configure Varnish to handle this. Since changing Magento is out of the question, I was wondering if someone can give me a clue as to how I would configure Varnish to handle this cookie?

Answer 1

http://moprea.ro/2011/may/6/magento-performance-optimization-varnish-cache-3/ describes the Magento extension that enables full page cache with varnish. This extension relies on Varnish config published on github.

These are the features already implemented:

1. Workable varnish config
2. Enable full page caching using Varnish, a super fast caching HTTP reverse proxy.
3. Varnish servers are configurable in Admin, under System / Configuration / General - Varnish Options
4. Automatically clears (only) cached pages when products, categories and CMS pages are saved.
5. Adds a new cache type in Magento Admin, under System / Cache Management and offers the possibility to deactivate the cache and refresh the cache.
6. Notifies Admin users when a category navigation is saved and Varnish cache needs to be refreshed so that the menu will be updated for all pages.
7. Turns off varnish cache automatically for users that have products in the cart or are logged in, etc.)
8. Default varnish configuration offered so that the module is workable. Screen shots: https://github.com/madalinoprea/magneto-varnish/wiki

Answer 2

How to cache Magento in Varnish (Theory) - There are 2 components to this

1) Static assets (eg. Images, CSS, JS) - This is a simple common pattern that involves detecting requests that belong to this category and setting a cache time (Or relying on the cache time being sent by the origin server) Example of this in gist form

2) HTML documents - This is the much more complex part of a good Magento solution. Its critical to cache HTML documents in Varnish to be able to improve Magento performance. HTML document generation is the most expensive (time consuming) thing that a Magento server will do.

The challenge with caching HTML documents comes from personalised content.

Personalised content and HTML documents

Magento, and all other ecommerce sites, manage the state of a particular user though a session. A session is a record of that particular user’s status on your site. This covers things such as: “Hello Bob” - at the top of the page “4 Things in Your Cart” - the status of your shopping cart on each page

These are items that cannot be shared amongst users and would cause a major problem should this happen (we call this “session leakage”).

How do we cache HTML pages if the HTML pages contain personalised information about who the person is and what is in their shopping cart?

There are 2 main methods of achieving this: Loading personalised elements of the page via additional requests after the page has loaded A common implementation method here is to use AJAX to request page elements that are personalised Leveraging a technology to mark components of the HTML document as cacheable and other’s uncachable (or un shareable amongst users). Varnish supports a technology called ESI (Edge Side Includes) that allows different parts of a HTML document to be cached differently.

Your Varnish implementation strategy must factor in user personalisation.

Implementation options for Varnish

Magento 1.X - The most widely used method for caching HTML documents in Magento version 1 is the open source product called Magento Turpentine (by Nexus). This is a plugin that is installed (via Magento Connect) and will automatically add ESI tags to your HTML documents so that Varnish can cache these resources. Magento Turpentine install / guide

Magento 2.X - The latest version of Magento (currently in beta) supports Varnish as its recommended solution for HTML caching in production. This is great news, Varnish is the recommended option from Magento and will work out of the box to improve your site speed.

How to make Varnish and Magento work well

Deployment is one thing, The next steps once you have a Varnish Magento solution implemented and working is to understand how well its performing. Getting metrics on cache hit rates and detailed logs on a request by request basis can be a challenge as it involves deploying a range of additional infrastructure (or being stuck doing manual log collection on a one off basis).

We have recently built this infrastructure to run Varnish as a service in the cloud (with full logs/metrics) - www.section.io - Plug aside this can be the most important element to actual success with you Varnish and Magento project as you need to be constantly tuning your implementation to manage things like varying query strings in urls (Hello google analytics "gclid"!) which can reduce your cache hit rates dramatically

Answer 3

If you are using Varnish 3.0, you may have to change your .vcl config. Here's what I am using with magento and varnish 3:

    # This is a basic VCL configuration file for varnish.  See the vcl(7)
# man page for details on VCL syntax and semantics.
# 
# Default backend definition.  Set this to point to your content
# server.
# 
 backend default {
    .host = "127.0.0.1";
    .port = "8080";
 }

acl trusted {
    "127.0.0.1";
    "127.0.1.1";
    # Add other ips that are allowed to purge cache
}

# 
# http://www.varnish-cache.org/docs/2.1/tutorial/vcl.html#vcl-recv
# @param req    Request object
sub vcl_recv {
    if (req.http.x-forwarded-for) {
        set req.http.X-Forwarded-For = req.http.X-Forwarded-For + ", " + client.ip;
    }
    else {
        set req.http.X-Forwarded-For = client.ip;
    }

    if (req.request == "PURGE") {
        # Allow requests from trusted IPs to purge the cache
        if (!client.ip ~ trusted) {
           error 405 "Not allowed.";
        }
        ban("req.url ~ " + req.url);
        error 200 "Ok"; #We don't go to backend 
        #return(lookup); # @see vcl_hit
    }

    if (req.request != "GET" &&
       req.request != "HEAD" &&
       req.request != "PUT" &&
       req.request != "POST" &&
       req.request != "TRACE" &&
       req.request != "OPTIONS" &&
       req.request != "DELETE") {
         /* Non-RFC2616 or CONNECT which is weird. */
         return (pipe);
    }

     # Cache only GET or HEAD requests
     if (req.request != "GET" && req.request != "HEAD") {
         /* We only deal with GET and HEAD by default */
         return (pass);
     }

    # parse accept encoding rulesets to normalize
    if (req.http.Accept-Encoding) {
        if (req.http.Accept-Encoding ~ "gzip") {
            set req.http.Accept-Encoding = "gzip";
        } elsif (req.http.Accept-Encoding ~ "deflate") {
            set req.http.Accept-Encoding = "deflate";
        } else {
            # unkown algorithm
            remove req.http.Accept-Encoding;
        }
    }

     # Rules for static files
     if (req.url ~ "\.(jpeg|jpg|png|gif|ico|swf|js|css|gz|rar|txt|bzip|pdf)(\?.*|)$") {
        set req.http.staticmarker = "1";
        unset req.http.Cookie;

        return (lookup);
    }

    # Don't cache pages for Magento Admin
    # FIXME: change this rule if you use custom url in admin
    if (req.url ~ "^/(index.php/)?admin") {
        return(pass);
    }

    # Don't cache checkout/customer pages, product compare
  #  if (req.url ~ "^/(index.php/)?(checkout|customer|catalog/product_compare|wishlist)") {
  #      return(pass);
  #  }
    # Don't cache checkout/customer pages, product compare
    if (req.url ~ "/(checkout|customer|catalog/product_compare|wishlist)/") {
        return(pass);
    }


    # Don't cache till session end
    if (req.http.cookie ~ "nocache_stable") {
        return(pass);
    }

    # Unique identifier witch tell Varnish use cache or not
    if (req.http.cookie ~ "nocache") {
        return(pass);
    }

    # Remove cookie 
    unset req.http.Cookie;
    set req.http.magicmarker = "1"; #Instruct varnish to remove cache headers received from backend
    return(lookup);
 }


sub vcl_pipe {
#     # Note that only the first request to the backend will have
#     # X-Forwarded-For set.  If you use X-Forwarded-For and want to
#     # have it set for all requests, make sure to have:
#     # set req.http.connection = "close";
#     # here.  It is not set by default as it might break some broken web
#     # applications, like IIS with NTLM authentication.
     return (pipe);
}

#sub vcl_pass {
#     return (pass);
#}

#sub vcl_hash {
#     set req.hash += req.url;
#     if (req.http.host) {
#         set req.hash += req.http.host;
#     } else {
#         set req.hash += server.ip;
#     }
#     return (hash);
# }


# Called after a cache lookup if the req. document was found in the cache.
sub vcl_hit {
    if (req.request == "PURGE") {
        ban_url(req.url);
        error 200 "Purged";
    }

    #
    # ATTENTION!! I had to comment this to make it work on vernish 3.0!!!!
    # error message:
    # Symbol not found: 'obj.cacheable' (expected type BOOL):
    #
    # I'm not sure about it, please check!!!
    #
    #if (!obj.cacheable) {
    #    return (pass);
    #}


    return (deliver);
}

# Called after a cache lookup and odc was not found in cache.
sub vcl_miss {
    if (req.request == "PURGE"){
        error 200 "Not in cache";
    }
    return (fetch);
}

# Called after document was retreived from backend
# @var req      Request object.
# @var beresp   Backend response (contains HTTP headers from backend)
sub vcl_fetch {
    set req.grace = 30s;

    # Current response should not be cached
    if(beresp.http.Set-Cookie ~ "nocache=1") {
        return (deliver);
    }

    # Flag set when we want to delete cache headers received from backend
    if (req.http.magicmarker){
        unset beresp.http.magicmarker;
        unset beresp.http.Cache-Control;
        unset beresp.http.Expires;
        unset beresp.http.Pragma;
        unset beresp.http.Cache;
        unset beresp.http.Server;
        unset beresp.http.Set-Cookie;
        unset beresp.http.Age;

        # default ttl for pages
        set beresp.ttl = 1d;
    }
    if (req.http.staticmarker) {
        set beresp.ttl = 30d; # static file cache expires in 30 days
        unset beresp.http.staticmarker;
        unset beresp.http.ETag; # Removes Etag in case we have multiple frontends
    }

    return (deliver);
}

# Called after a cached document is delivered to the client.
sub vcl_deliver {
    if (obj.hits > 0) {
        set resp.http.X-Cache = "HIT ("+obj.hits+")";
    } else {
        set resp.http.X-Cache = "MISS";
        #    set resp.http.X-Cache-Hash = obj.http.hash;
    }
    return (deliver);
}
# 
# sub vcl_error {
#     set obj.http.Content-Type = "text/html; charset=utf-8";
#     synthetic {"
# <?xml version="1.0" encoding="utf-8"?>
# <!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Strict//EN"
#  "http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd">
# <html>
#   <head>
#     <title>"} obj.status " " obj.response {"</title>
#   </head>
#   <body>
#     <h1>Error "} obj.status " " obj.response {"</h1>
#     <p>"} obj.response {"</p>
#     <h3>Guru Meditation:</h3>
#     <p>XID: "} req.xid {"</p>
#     <hr>
#     <address>
#        <a href="http://www.varnish-cache.org/">Varnish cache server</a>
#     </address>
#   </body>
# </html>
# "};
#     return (deliver);
# }

Answer 4

I am assuming that's a session cookie that Magento sends to all users - I had a similar problem with Varnish + Redmine.

The reason Varnish is not caching your pages is because by default it only caches what it is sure is safe - and users with cookies typically see different things for a given page load, for example if they're logged in then maybe their username is at the top of each page, so the page can't be cached†.

Many frameworks however give session cookies to users that aren't logged in as well. I'm afraid I don't know Magento at all so I cannot predict the consequences of ignoring this cookie - on Redmine, ignoring the cookie meant that users could not log in, and all forms stopped working (because they no longer had the CSRF token).

It would probably be better to tackle this from the Magento side if you can - Varnish will listen to the upstream's headers to determine what can be cached etc.

If you can't, then you might be able to mitigate it from Varnish's configuration. You will want to ensure that the Set-Cookie header is not sent from any cache hit, and you will also want to drop the client's cookie on requests for pages where that cookie has no effect. This means you will need exceptions for things such as the login screen, or any page that requires you to be logged in (unless Magento sets a separate cookie once you are logged in, which would make things a lot easier).

The Varnish documentation (which I can highly recommend as a resource) has several pages on increasing hit rate, including one specifically for dropping cookies on some pages and not others.

† There is an exception, which is if you are using edge side includes.

Answer 5

This i think explains how we could away with using varnish with magento

If you use aoe_static module and my custom vcl for varnish 3 it clears the cookies on the cached page response. It should do this in vcl fetch. The cookies can then be set from a smaller ajax response which loads the dynamic content. This maintains your sessions, cart etc. This ajax response can be "pipe"ed in vcl recovery.

I haven't experienced any problems doing this but haven't tried it on a production site.

Dynamic blocks must be replaced with placeholders via layout xml. If you liked these replacements could be varnish edge side includes or custom ajax implementation.

When loading the dynamic content from aoe_static (or whatever type ajax methods you prefer) it's good to remember that you can still use magentos layout system, e.g. create a handle for your ajax call with nested blocks to be rendered.

If you use the aoe_static module you'll notice that the loadLayout is called but remember what handles are passed to that loadLayout. it's not the same as a layout request for the page your on but it does get a default handle for you.

Another problem is stock levels. When a product no longer has a enough stock to be added to cart it will still appear on product lists and as options for configurable and grouped products.

Possibly you could use the observer - cataloginventory_stock_item_save_after - to check stock levels (i haven't checked this). Then cache could be purged based on the url of the products. It's quite easy to get the category urls that the product would appear in and purge these at the same time.

The phoenix module has methods do these sorts of purges if you want to see an easy implementation just follow through from observers.

But how to deal with the layered nav urls is more tricky. you would need to store the query string parameters that the app had served using the base category list url as a key in advance then read and purge these urls in the observer. This storing of query string parameters would be easy enough using the response before send,checking the request object with a regex and logging the query strings used comma separated.

I am wrong in thinking that none of the current modules deal with stock levels for layered nav?

I think a good finished module for varnish is needed in the open source community since all other are lacking. personally i plan to only use paid full page cache's with load balanced servers and maybe varnish to catch image and css requests. Unless someone wants to join forces to make a proper varnish implementation or I would be happy to assist with issues for your sites if the work could be added to an open source implementation that deals better with all these concerns.

checkout this question for more details on problems you will face this question - magento open source full page cache

Answer 6

We have developed a module called PageCache powered by Varnish that enables Magento and Varnish to play smoothly together by providing a well tested Varnish configuration file (VCL) and a tightly integrated Magento module with plenty of options to control Varnish rigth from the Magento backend. Check it out on Magento Connect:

http://www.magentocommerce.com/magento-connect/Phoenix/extension/6322/varnish_cache

Answer 7

http://moprea.ro/2011/feb/16/magento-performance-optimization-varnish-cache-2/

Not sure if it helps but I chanced upon this.

I have actually tried out getting varnish to work before and I have failed. I would suggest you get APC + Memcached + tmpfs sessions/cache before trying out varnish.