3

Is it possible to write to Redis in parallel from spark?

(Or: how to write tens of thousands of keys/lists quickly from spark)

Currently, I'm writing to Redis by key in sequence, and it's taking forever. I need to write about 90000 lists (of length 2-2000). Speed is extremely important. Currently, it's taking on the order of 1 hour. Tradition benchmarks of Redis claim thousands of Redis writes per second, but in my pipeline, I'm not anywhere near that.

Any help is appreciated.

Guy Korland
  • 9,139
  • 14
  • 59
  • 106
BBischof
  • 310
  • 2
  • 13
  • 1
    As I asked you on twitter - what is your setup? A single redis? A redis cluster? RLEC? – Not_a_Golfer Apr 15 '16 at 13:28
  • Just using a single Redis actually. Running in a mesos cluster. – BBischof Apr 15 '16 at 18:15
  • The connector is optimized to run over a redis cluster where every spark node has a local redis node, and the partitioning functions match so that you'll always write to local redis. How many spark nodes are you running? – Not_a_Golfer Apr 15 '16 at 19:28
  • @Not_a_Golfer four nodes. I suppose we could try running a redis local to each node. How would we read keys from a single ip? We could do a rebalance probably. – BBischof Apr 15 '16 at 22:58
  • Do you use pipeline for sending command to Redis? Pipeline is very fast compared to one by one sending commands. – halil Oct 25 '16 at 13:32

1 Answers1

2

A single Redis instance runs in one thread so operations are inherently sequential. If you have a Redis cluster then the instance to which a datum is written depends on a hash slot calculated from the key being written. This hash function (amongst other things) ensures that the load gets distributed across all the Redis instances in the cluster. If you cluster has N instances, you have (almost) at most N parallel writes that you can execute. This because each cluster instance is still a single thread. A reasonable Spark Redis connector should exploit the cluster efficiently.

Either way, Redis is really quick, especially if you use mass inserts.

David Weber
  • 1,965
  • 1
  • 22
  • 32