0

I was looking for a way to share object in several nodes cluster, and after a bit of research I thought it would be best using redis pub/sub. Then, I saw that redis doesn't support cluster yet, which means that a system based on redis will have single point of failure. Since high availability is a key feature for me, this solution is not applicable.

At the moment, I am looking into 2 other solutions for this issue:

I have 2 questions:

  1. On top of which solution it would be more efficient to simulate pub/sub?
  2. which is better when keeping clusters in mind?

I was hoping that someone out there faced similar issues and share his experience.

Community
  • 1
  • 1
Kuf
  • 17,318
  • 6
  • 67
  • 91
  • 1
    I don't understand how using memcached wouldn't be a single point of failure? It's an in-memory only cache. I wouldn't consider that "highly available" as data could be lost at any time. I'd suggest you look for actual pub-sub systems if you need something robust and efficient. – WiredPrairie Jun 29 '13 at 14:09
  • memchached is indeed in memory but i save my object anyway after each change, so memchached cluster should work. My main issue is the pub/sub ability. Is there a solution similar to redis pub/sub that supports clustering? – Kuf Jun 29 '13 at 15:06
  • I don't follow still ... what are you using for HA for memcached? Are you planning on trying to replicate data between memcached nodes? Again, I think you're using the wrong tool for this requirement. When an instance of memcached dies, along with all of the data, how will your code know which things were "published" to the subscribers successfully? – WiredPrairie Jun 29 '13 at 15:26
  • I am looking for a way to pass messages between node.js servers. I am looking for an elastic solution that will send the message to all other nodes, while update dynamically the list of active servers. If you can think of a better way to achieve this i would appreciate if you'll share it with me – Kuf Jun 29 '13 at 15:49
  • How many servers / how many messages / how "dynamic" is the list? – WiredPrairie Jun 29 '13 at 16:29
  • for now up to 100k messages per seconds, but with the option to grow to millions. the amount of servers will be dynamically derived from servers load. so, list is updated as the load grows bigger or smaller. – Kuf Jun 29 '13 at 16:39
  • I think with such volume you should really look to actual pub-sub apps, as @WiredPrairie said. As far as I remember memcached "pub-sub" provider generates ~150-200 ops/sec even in idle state. And with bigger number of servers this load will only grow. Guys from [moot.it](https://moot.it/blog/technology/redis-as-primary-datastore-wtf.html) use redis and I think they have more than 1 server. – m03geek Jun 29 '13 at 16:59

1 Answers1

1

  1. I think it's a bad idea to use memcached and couchbase for pub/sub. Both solutions don't provide built-in pub/sub functions and implementing pub/sub on app side can cause a lot of ops/sec to memcache/couchbase server and as a result you'll get slow performance. Couchbase stores data into disk, so for temporary storage it's better to use memcaced. It will be faster and will not load your disk. If you can avoid that "pub/sub" and use memcached/couchbase just as simple HA shared key-value storage - do it. It will be much better then pub/sub.

  2. When you install Couchbase server it provides 2 types of buckets: couchbase (with disk persistance, ability to create views, etc.) and memcached (only in-memory key-value storage). Both types of buckets act in the same way in clusters. Also couchbase support memcache api calls, so you don't need to change code to test both variants.

I've tried to use memcached provider for socket.io "pub/sub" sharing, but as I mentioned before it's ugly. And in my case there were few node.js servers with socket.io, so instead of sharing I've implemented something like "p2p messaging" between servers on top of sockets.

UPD: If you have such big amount of data may be it will be better not to have one shared storage, but use something like sharding with "predictible" data location.

m03geek
  • 2,508
  • 1
  • 21
  • 41
  • The reason I wanted pub/sub in the first place was to avoid the p2p messaging as well as the complications that might arise from using it. for example, how can you keep the dynamic list of active servers that needs to be notified? – Kuf Jun 29 '13 at 15:06
  • I've had constant number (3) of servers with fixed ips, so I was able to use "p2p". I had copy of list on each node and send only changes (add/remove). If you have a lot of servers or need to add/remove them dynamicly - pub/sub is only one variant. You can try memcached pub/sub provider for node to test it. – m03geek Jun 29 '13 at 16:28