0

I wrote a script and ran it on the darwin system. In order to maximize the efficiency of my script (bypassing the GIL), I wrote the code with the following structure:

import multiprocessing as mp
import threading
import queue


p_num = 2
t_num = 13


def fill_queue(q):
    for i in range(10000):
        q.put(i)


def do_sth(job: int):
    # print(job)
    return

def run_in_thread(q):
    while True:
        try:
            job = q.get(block=False)
        except queue.Empty:
            return
        else:
            try:
                do_sth(job)
            except BaseException as e:
                raise e
            finally:
                continue


def run_in_proc(q):
    threads = [threading.Thread(target=run_in_thread, args=(q,)) for _ in range(t_num)]
    for thread in threads:
        thread.start()
    for thread in threads:
        thread.join()


if __name__ == "__main__":
    manager = mp.Manager()
    q = manager.Queue()
    fill_queue(q)
    procs = [mp.Process(target=run_in_proc, args=(q,)) for _ in range(p_num)]
    for proc in procs:
        proc.start()
    for proc in procs:
        proc.join()

Then I got the following exception from q.get():

Exception in thread Thread-11:
Traceback (most recent call last):
  File "/Users/my_name/.pyenv/versions/3.8.10/lib/python3.8/multiprocessing/managers.py", line 827, in _callmethod
    conn = self._tls.connection
AttributeError: 'ForkAwareLocal' object has no attribute 'connection'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/my_name/.pyenv/versions/3.8.10/lib/python3.8/threading.py", line 932, in _bootstrap_inner
    self.run()
  File "/Users/my_name/.pyenv/versions/3.8.10/lib/python3.8/threading.py", line 870, in run
    self._target(*self._args, **self._kwargs)
  File "/Users/my_name/century/Easy/temp/demo.py", line 22, in run_in_thread
    job = q.get(block=False)
  File "<string>", line 2, in get
  File "/Users/my_name/.pyenv/versions/3.8.10/lib/python3.8/multiprocessing/managers.py", line 831, in _callmethod
    self._connect()
  File "/Users/my_name/.pyenv/versions/3.8.10/lib/python3.8/multiprocessing/managers.py", line 818, in _connect
    conn = self._Client(self._token.address, authkey=self._authkey)
  File "/Users/my_name/.pyenv/versions/3.8.10/lib/python3.8/multiprocessing/connection.py", line 502, in Client
    c = SocketClient(address)
  File "/Users/my_name/.pyenv/versions/3.8.10/lib/python3.8/multiprocessing/connection.py", line 630, in SocketClient
    s.connect(address)
ConnectionRefusedError: [Errno 61] Connection refused

I tried to modify the system parameter kern.ipc.somaxconn of darwin:

sudo sysctl -w kern.ipc.somaxconn=102400

I also tried to trace the source code, but there is a section of the stack that is not implemented by python. I can't figure out the cause of this exception and how I should avoid it. I hope someone can answer this question for me, I realy appreciate.

m276
  • 1
  • 1
  • I have been reminded to view this issue :https://stackoverflow.com/questions/60049527/shutting-down-manager-error-attributeerror-forkawarelocal-object-has-no-attr and I found the answer in @Programmieru s' answer – m276 Jan 27 '23 at 08:48

0 Answers0