How redis pipe-lining works in pyredis?

Question

I am trying to understand, how pipe lining in redis works? According to one blog I read, For this code

Pipeline pipeline = jedis.pipelined();
long start = System.currentTimeMillis();
for (int i = 0; i < 100000; i++) {
    pipeline.set("" + i, "" + i);
}
List<Object> results = pipeline.execute();

Every call to pipeline.set() effectively sends the SET command to Redis (you can easily see this by setting a breakpoint inside the loop and querying Redis with redis-cli). The call to pipeline.execute() is when the reading of all the pending responses happens.

So basically, when we use pipe-lining, when we execute any command like set above, the command gets executed on the server but we don't collect the response until we executed, pipeline.execute().

However, according to the documentation of pyredis, Pipelines are a subclass of the base Redis class that provide support for buffering multiple commands to the server in a single request.

I think, this implies that, we use pipelining, all the commands are buffered and are sent to the server, when we execute pipe.execute(), so this behaviour is different from the behaviour described above.

Could someone please tell me what is the right behaviour when using pyreids?

score 3 · Answer 1 · answered Oct 22 '15 at 23:33

This is not just a redis-py thing. In Redis, pipelining always means buffering a set of commands and then sending them to the server all at once. The main point of pipelining is to avoid extraneous network back-and-forths-- frequently the bottleneck when running commands against Redis. If each command were sent to Redis before the pipeline was run, this would not be the case.

You can test this in practice. Open up python and:

import redis
r = redis.Redis()
p = r.pipeline()
p.set('blah', 'foo') # this buffers the command. it is not yet run.
r.get('blah') # pipeline hasn't been run, so this returns nothing.
p.execute()
r.get('blah') # now that we've run the pipeline, this returns "foo".

score 0 · Answer 2 · edited Jun 20 '20 at 09:12

I did run the test that you described from the blog, and I could not reproduce the behaviour. Setting breakpoints in the for loop, and running

redis-cli info | grep keys

does not show the size increasing after every set command.

Speaking of which, the code you pasted seems to be Java using Jedis (which I also used). And in the test I ran, and according to the documentation, there is no method execute() in jedis but an exec() and sync() one.

I did see the values being set in redis after the sync() command.

Besides, this question seems to go with the pyredis documentation.

Finally, the redis documentation itself focuses on networking optimization (Quoting the example)

This time we are not paying the cost of RTT for every call, but just one time for the three commands.

P.S. Could you get the link to the blog you read?

How redis pipe-lining works in pyredis?

2 Answers2