Installed Redis Cluster with Twemproxy, I am really confused about why some SET commands are MOVED

Question

I have setup a Redis-Cluster with version 3.0.5 of Redis-Server (Ubuntu 14.04)

For simplicity, we shall ignore replication. I have three redis instances running on localhost, ports 7001, 7002 and 7003. They are all made masters of a cluster with this command

redis-trib.rb create 127.0.0.1:7001 127.0.0.1:7002 127.0.0.1:7003

I like the idea of using twemproxy

twemproxy-config.yml

twem-1:
  listen: 127.0.0.1:22121
  hash: fnv1a_64
  distribution: ketama
  redis: true
  preconnect: true
  servers:
  - 127.0.0.1:7001:1
  - 127.0.0.1:7002:1
  - 127.0.0.1:7003:1

I initialize twemproxy with nutcracker -c twemproxy-config.yml -d and then I can access twemproxy with redis-cli -h 127.0.0.1 -p 22121

Please have a look at this input and output

127.0.0.1:22121> set hello 4542342342424
OK
127.0.0.1:22121> set goodbye 345353535545
(error) MOVED 9354 159.203.136.204:7002

127.0.0.1:22121> get hello
"4542342342424"
127.0.0.1:22121> get goodbye
(error) MOVED 9354 159.203.136.204:7002

I am concerned that this may not be working properly. If I bypass twemproxy and connect using redis-cli -c -h 127.0.0.1 -p 7001, I can see automatic forwarding taking place. Like so;

127.0.0.1:7001> get hello
"4542342342424"

127.0.0.1:7001> get goodbye
-> Redirected to slot [9354] located at 127.0.0.1:7002
(nil)

127.0.0.1:7002> set goodbye 3240923842094840
OK

127.0.0.1:7002> get goodbye
"3240923842094840"

Reference

Redis Cluster Specification

Interesting read at code.hootsuite.com about use of twemproxy (about halfway down page)

Ultimate aim

My ultimate aim is to use the redis cluster for storing PHP session data across multiple webservers behind a load balancer. In php.ini I would have session.save_handler = redis and session.save_path = tcp://127.0.0.1:22121 (the twemproxy instance would run on each web server). The PHP session configuration bit isn't set up yet.

I hope I am making sense. Am I using the correct hashing code? I really would like to see twemproxy giving back OK instead of MOVED.

Thanks!

Update

Thanks to the answer from @the-real-bill I have setup redis on two nodes with standard redis running port 6380 and sentinel running port 16380, one is master the other is slave. Watching the logs, everything looks good.

I've looked at predis and phpredis

This is the bit I'm still a little unsure about and I am sure given more time I will work it out. Queries can only be written to a master, and queries can only be read from a slave. We've got Sentinel promoting one or the other depending on availabbility - how does the session handler know which one it can write to? Surely I need to provide two IP adddresses..

score 3 · Accepted Answer · answered Nov 02 '15 at 21:54

Redis Cluster is a client-aware pattern. In Redis Cluster the client always most connect to the "correct" node for a given key. The MOVED reply is telling the client which node services that key. Initially a client is expected to pull the current topology on connect and then update if it gets a MOVED. This is the highest performing pattern as there is no proxy involved.

However, this means you can not use any existing proxy in front of a Redis Cluster setup. As you have seen Twemproxy simply proxies certain commands, and refuses others. In order for Twemproxy to handle this it would need to implement the cluster API and do everything the client does. This is does not do, nor should you expect it to do such soon. Any other proxy such as nginx or HAProxy would need to do the same thing (hypothetically with openresty or a custom module you could do this in Nginx easier than the others).

Further, last I checked the PHP clients do not support Redis Cluster.

However, for your use case I doubt you actually need a Redis Cluster. Configuring Redis in a Pod (master+slave(s)) with Sentinel to manage failover combined with client-side Sentinel support would likely meet your needs quite well. I believe PRedis has support but I am sure phpredis does not.

If you have your heart set on the complexity of running a shared cluster, then use regular Redis + Sentinel behind Twemproxy and let Twemproxy connect via the sharing algorithm you tell it to use. This is the other rub with trying to use Tweproxy - it does what Redis Cluster does but as a proxy. So in effect you'd be trying to shard the data set twice. I' sure you can imagine ways in which that is catastrophic. ;)

If you can't get proper support or don't want Twemproxy managing sharing for you, your final option (short of adding client support yourself) is to set up a proxy which is configured by sentinel. For example you could use HAProxy configured to point to the master. On a failover Sentinel can execute a script for you.

This script could then update your HAProxy configuration and restart it for you, thus ensuring proper reconnect behavior. There are many ways to do this and a Google search on it, combined with the Redis Sentinel docs and applying them to your environment would show the path for you.

I actually came across your page at http://www.iamtherealbill.com/2015/04/clusterizing-redis-intro yesterday before you posted this. Your answer above certainly does put the cream on top and helps me understand better. You are right I do not need Redis Cluster, I should look at master+slave with failover. I certainly do not want anything catastrophic - I just want two webservers containing duplicate websites to access a shared session hold, and obviously want that to be highly available too. I'll look into further reading about Sentinel. — anastymous, Nov 03 '15 at 07:10

score 1 · Answer 2 · answered Nov 02 '15 at 14:37

You can not use Redis cluster and twemproxy at the same time. In Redis cluster mode, it should be the client to send the request to the right master. When you request a key from a wrong master, it will always send back a MOVE reply, redirecting you to the right master.

Installed Redis Cluster with Twemproxy, I am really confused about why some SET commands are MOVED

Update

2 Answers2