3

I understand that memcached is a distributed caching system. However, is it entirely necessary for memcached to replicate? The objective is to persist sessions in a clustered environment.

For example if we have memcached running on say 2 servers, both with data on it, and server #1 goes down, could we potentially lose session data that was stored on it? In other words, what should we expect to see happen should any memcached server (storing data) goes down and how would it affect our sessions in a clustered environment?

At the end of the day, will it be up to use to add some fault tolerance to our application? For example, if the key doesn't exist possibly because one of the servers it was on went down, re-query and store back to memcached?

From what I'm reading, it appears to lean in this direction but would like confirmation: https://developers.google.com/appengine/articles/scaling/memcache#transient

Thanks in advance!

Matt Borja
  • 1,509
  • 1
  • 17
  • 38

2 Answers2

3

Memcached has it's own fault tolerance built in so you don't need to add it to your application. I think providing an example will show why this is the case. Let's say you have 2 memcached servers set up in front of your database (let's say it's mysql). Initially when you start your application there will be nothing in memcached. When your application needs to get data if will first check in memcached and if it doesn't exist then it will read the data from the database and insert it into memcached before returning it to the user. For writes you will make sure that you insert the data into both your database and memcached. As you application continues to run it will populate the memcached servers with a bunch of data and take load off of your database.

Now one of your memcached servers crashes and you lose half of your cached data. What will happen is that your application will now be going to the database more frequently right after the crash and your application logic will continue to insert data into memcached except everything will go directly to the server that didn't crash. The only consequence here is that your cache is smaller and your database might need to do a little bit more work if everything doesn't fit into the cache. Your memcached client should also be able to handle the crash since it will be able to figure out where your remaining healthy memcached servers are and it will automatically hash values into them accordingly. So in short you don't need any extra logic for failure situations in memcached since the memcached client should take care of this for you. You just need to understand that memcached servers going down might mean your database has to do a lot of extra work. I also wouldn't recommend re-populating the cache after a failure. Just let the cache warm itself back up since there's no point in loading items that you aren't going to use in the near future.

m03geek also made a post where he mentioned that you could also use Couchbase and this is true, but I want to add a few things to his response about what the pros and cons are. First off Couchbase has two bucket (database) types and these are the Memcached Bucket and the Couchbase Bucket. The Memcached bucket is plain memcached and everything I wrote above is valid for this bucket. The only reasons you might want to go with Couchbase if you are going to use the memcached bucket are that you get a nice web ui which will provide stats about your memcached cluster along with ease of use of adding and removing servers. You can also get paid support down the road for Couchbase.

The Couchbase bucket is totally different in that it is not a cache, but an actual database. You can completely drop your backend database and just use this bucket type. One nice thing about the Couchbase bucket is that it provides replication and therefore prevents the cold cache problem that memcached has. I would suggest reading the Couchbase documentation if this sounds interesting you you since there are a lot of feature you get with the Couchbase bucket.

This paper about how Facebook uses memcached might be interesting too. https://www.usenix.org/system/files/conference/nsdi13/nsdi13-final170_update.pdf

mikewied
  • 5,273
  • 1
  • 20
  • 32
  • Thanks, we are actually using Couchbase and are currently evaluating its performance against our application. As far as fault tolerance goes, we do have a working solution in place that updates both memcache and database and calls from memcache first. Could you please elaborate on what you mean by "allowing memcache to warm up" though? From what I understand, our application would still have to re-populate cache with missing variables from a downed memcache server if such occurred. – Matt Borja Sep 13 '13 at 20:49
  • Your application Repopulating the cache is what I meant by warming up. – mikewied Sep 18 '13 at 19:37
0

Couchbase embedded memcached and "vanilla" memcached have some differences. One of them, as far as I know, is that couchbase's memcached servers act like one. This means that if you store your key-value on one server, you'll be able to retreive it from another server in cluster. And vanilla memcached "clusters" are usally built with sharding technique, which means on app side you should know what server contain desired key.

My opinion is that replicating memcached data is unnessesary. Modern datacenters provide almost 99% uptime. So if someday one of your memcached servers will go down just some of your online users will be needed to relogin.

Also on many websites you can see "Remember me" checkbox that sets a cookie, which can be used to restore session. If your users will have that cookie they will not even notice that one of your servers were down. (that's answer for your question about "add some fault tolerance to our application")

But you can always use something like haproxy and replicate all your session data on 2 or more independent servers. In this case to store 1 user session you'll need N times more RAM, where N is number of replicas.

Another way - to use couchbase to store sessions. Couchbase cluster support replicas "out of the box" and it also stores data on disk, so if your node (or all nodes) will suddenly shutdown or reboot, session data will not lost.

Short answer: memcached with "remember me" cookie and without replication should be enough.

m03geek
  • 2,508
  • 1
  • 21
  • 41
  • 1
    Not sure what is being implied here when you say the "Remember me" cookie can be used to restore a session. Restore it from what? A database query using that cookie value? – Matt Borja Sep 12 '13 at 19:13
  • Usally session contains some information about user's account that is stored somewhere in database. When user logges in this information is retreived form DB to session. "Remember me" cookie stores some token, that could identify user without making him to login (input login and pass). So if user is not logged in (session doesn't contains any info about system account) and this cookie is present, then you get user's account by that cookie token and writes it into session. See http://stackoverflow.com/questions/244882/what-is-the-best-way-to-implement-remember-me-for-a-website. – m03geek Sep 12 '13 at 20:55
  • Okay, that's what I thought you were implying. We are pretty much doing this at this point but memcaching variables so as to reduce seemingly expensive queries to database. – Matt Borja Sep 13 '13 at 20:46