I wrote a script and ran it on the darwin system. In order to maximize the efficiency of my script (bypassing the GIL), I wrote the code with the following structure:
import multiprocessing as mp
import threading
import queue
p_num = 2
t_num = 13
def fill_queue(q):
for i in range(10000):
q.put(i)
def do_sth(job: int):
# print(job)
return
def run_in_thread(q):
while True:
try:
job = q.get(block=False)
except queue.Empty:
return
else:
try:
do_sth(job)
except BaseException as e:
raise e
finally:
continue
def run_in_proc(q):
threads = [threading.Thread(target=run_in_thread, args=(q,)) for _ in range(t_num)]
for thread in threads:
thread.start()
for thread in threads:
thread.join()
if __name__ == "__main__":
manager = mp.Manager()
q = manager.Queue()
fill_queue(q)
procs = [mp.Process(target=run_in_proc, args=(q,)) for _ in range(p_num)]
for proc in procs:
proc.start()
for proc in procs:
proc.join()
Then I got the following exception from q.get():
Exception in thread Thread-11:
Traceback (most recent call last):
File "/Users/my_name/.pyenv/versions/3.8.10/lib/python3.8/multiprocessing/managers.py", line 827, in _callmethod
conn = self._tls.connection
AttributeError: 'ForkAwareLocal' object has no attribute 'connection'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/Users/my_name/.pyenv/versions/3.8.10/lib/python3.8/threading.py", line 932, in _bootstrap_inner
self.run()
File "/Users/my_name/.pyenv/versions/3.8.10/lib/python3.8/threading.py", line 870, in run
self._target(*self._args, **self._kwargs)
File "/Users/my_name/century/Easy/temp/demo.py", line 22, in run_in_thread
job = q.get(block=False)
File "<string>", line 2, in get
File "/Users/my_name/.pyenv/versions/3.8.10/lib/python3.8/multiprocessing/managers.py", line 831, in _callmethod
self._connect()
File "/Users/my_name/.pyenv/versions/3.8.10/lib/python3.8/multiprocessing/managers.py", line 818, in _connect
conn = self._Client(self._token.address, authkey=self._authkey)
File "/Users/my_name/.pyenv/versions/3.8.10/lib/python3.8/multiprocessing/connection.py", line 502, in Client
c = SocketClient(address)
File "/Users/my_name/.pyenv/versions/3.8.10/lib/python3.8/multiprocessing/connection.py", line 630, in SocketClient
s.connect(address)
ConnectionRefusedError: [Errno 61] Connection refused
I tried to modify the system parameter kern.ipc.somaxconn of darwin:
sudo sysctl -w kern.ipc.somaxconn=102400
I also tried to trace the source code, but there is a section of the stack that is not implemented by python. I can't figure out the cause of this exception and how I should avoid it. I hope someone can answer this question for me, I realy appreciate.