I did a performance test on multiprocessiong pipe, and I found that pipe is much faster in python2 than python3. Python3 is about four times slower than python2. I am worried about such a large performance loss. What caused this problem? Below are my test results.
CentOS Linux release 7.4.1708
[root@localhost tests]# python --version
Python 2.7.5
[root@localhost tests]# python test.py pipe
sending 10000 message to pipe cost 0.0198199748993s
sending 100000 message to pipe cost 0.212320804596s
sending 1000000 message to pipe cost 1.94675683975s
[root@localhost tests]# python3 --version
Python 3.6.8
[root@localhost tests]# python3 test.py pipe
sending 10000 message to pipe cost 0.10193729400634766s
sending 100000 message to pipe cost 0.7190062999725342s
sending 1000000 message to pipe cost 6.87559700012207s
Windows 10
C:\Users\tests>py -2 --version
Python 2.7.18
C:\Users\tests>py -2 test.py pipe
sending 10000 message to pipe cost 0.121000051498s
sending 100000 message to pipe cost 0.535000085831s
sending 1000000 message to pipe cost 3.50300002098s
C:\Users\tests>py -3 --version
Python 3.7.6
C:\Users\tests>py -3 test.py pipe
sending 10000 message to pipe cost 0.2459118366241455s
sending 100000 message to pipe cost 1.256772518157959s
sending 1000000 message to pipe cost 10.282079935073853s
Source Code:
import sys
import time
import multiprocessing
def reader(channel):
try:
recv = channel.get
except AttributeError:
recv = channel[0].recv
while True:
msg = recv()
if msg is None:
break
def writer(count, channel):
try:
send = channel.put
except AttributeError:
send = channel[1].send
for i in range(0, count):
send(i)
send(None)
if __name__ == '__main__':
arg = sys.argv[1]
if arg == "pipe":
channel = multiprocessing.Pipe()
else:
channel = multiprocessing.Queue()
for count in [10 ** 4, 10 ** 5, 10 ** 6]:
reader_p = multiprocessing.Process(target=reader, args=(channel,))
reader_p.daemon = True
reader_p.start()
start = time.time()
writer(count, channel)
reader_p.join()
print("sending {} message to {} cost {}s".format(count, arg, time.time()-start))