0

I am following the suggestions from these stackoverflow issues [1][2]. I have my multiprocessing function in a different module. I call it from main. The goal is to read images from multiple processes. I wait for all the children to finish and join. I am using Ubuntu 18.04. Any suggestions to debug this is appreciated.

Edit: here's the complete stacktrace

Traceback (most recent call last):
  File "/usr/local/lib/python3.6/multiprocessing/managers.py", line 749, in _callmethod
    conn = self._tls.connection
AttributeError: 'ForkAwareLocal' object has no attribute 'connection'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "main.py", line 98, in <module>
    input_transforms, model)
  File "/home/user/visualize_model/src/utils.py", line 126, in create_embedding_parallel
    input_images = list(input_images)
  File "<string>", line 2, in __len__
  File "/usr/local/lib/python3.6/multiprocessing/managers.py", line 753, in _callmethod
    self._connect()
  File "/usr/local/lib/python3.6/multiprocessing/managers.py", line 740, in _connect
    conn = self._Client(self._token.address, authkey=self._authkey)
  File "/usr/local/lib/python3.6/multiprocessing/connection.py", line 487, in Client
    c = SocketClient(address)
  File "/usr/local/lib/python3.6/multiprocessing/connection.py", line 614, in SocketClient
    s.connect(address)
FileNotFoundError: [Errno 2] No such file or directory

main.py

from utils import create_embeddings

if __name__ == "__main__":
    embeddings = utils.create_embeddings(folder, num_frames)
    # more statements follow after this

utils.py

from PIL import Image
import os
import multiprocessing as mp

def f(image_list, folder, index):
    image_path = os.path.join(folder, "images{:04d}.png".format(index))
    image_list.append(Image.open(image_path))

def create_embeddings(folder, num_frames):

    with mp.Manager() as manager:
        image_list = manager.list()
        processes = []
        for index in range(num_frames):
            p = mp.Process(target=f, args=(image_list, folder, index))
            p.start()
            processes.append(p)
        for p in processes:
            p.join()
        
    return image_list
Adda
  • 23
  • 3
  • Shouldn't you be more worried about the other error, `AttributeError: 'ForkAwareLocal' object has no attribute 'connection'`, since it happened first? – CryptoFool Sep 17 '20 at 00:29
  • This SO question involves that same error: https://stackoverflow.com/questions/43903884. It's closes as a duplicate, so you have a whole other SO question that might help also, but that one doesn't mention that specific error. – CryptoFool Sep 17 '20 at 00:33
  • Hi @Steve, I am not letting the manager die before I join the processes. In the other SO links, the issue it seems was join was missing. – Adda Sep 17 '20 at 01:01

0 Answers0