2

I did a performance test on multiprocessiong pipe, and I found that pipe is much faster in python2 than python3. Python3 is about four times slower than python2. I am worried about such a large performance loss. What caused this problem? Below are my test results.

CentOS Linux release 7.4.1708

[root@localhost tests]# python --version
Python 2.7.5
[root@localhost tests]# python test.py pipe
sending 10000 message to pipe cost 0.0198199748993s
sending 100000 message to pipe cost 0.212320804596s
sending 1000000 message to pipe cost 1.94675683975s
[root@localhost tests]# python3 --version
Python 3.6.8
[root@localhost tests]# python3 test.py pipe
sending 10000 message to pipe cost 0.10193729400634766s
sending 100000 message to pipe cost 0.7190062999725342s
sending 1000000 message to pipe cost 6.87559700012207s

Windows 10

C:\Users\tests>py -2 --version
Python 2.7.18

C:\Users\tests>py -2 test.py pipe
sending 10000 message to pipe cost 0.121000051498s
sending 100000 message to pipe cost 0.535000085831s
sending 1000000 message to pipe cost 3.50300002098s

C:\Users\tests>py -3 --version
Python 3.7.6

C:\Users\tests>py -3 test.py pipe
sending 10000 message to pipe cost 0.2459118366241455s
sending 100000 message to pipe cost 1.256772518157959s
sending 1000000 message to pipe cost 10.282079935073853s

Source Code:

import sys
import time
import multiprocessing


def reader(channel):
    try:
        recv = channel.get
    except AttributeError:
        recv = channel[0].recv
    while True:
        msg = recv()
        if msg is None:
            break


def writer(count, channel):
    try:
        send = channel.put
    except AttributeError:
        send = channel[1].send
    for i in range(0, count):
        send(i)
    send(None)


if __name__ == '__main__':
    arg = sys.argv[1]
    if arg == "pipe":
        channel = multiprocessing.Pipe()
    else:
        channel = multiprocessing.Queue()

    for count in [10 ** 4, 10 ** 5, 10 ** 6]:
        reader_p = multiprocessing.Process(target=reader, args=(channel,))
        reader_p.daemon = True
        reader_p.start()
        start = time.time()
        writer(count, channel)
        reader_p.join()
        print("sending {} message to {} cost {}s".format(count, arg, time.time()-start))
ningod
  • 21
  • 1
  • 1
    Related: [Python 3.4 multiprocessing Queue faster than Pipe, unexpected](https://stackoverflow.com/questions/26908909/python-3-4-multiprocessing-queue-faster-than-pipe-unexpected) – Selcuk Jun 07 '21 at 04:58

0 Answers0