Python multiprocessing memory allocation too high

Question

I have already read read some posts about my problem (like this one), but none of them could solve the issue. In my Flask App I download 32-64 images in parallel using Python built in multiprocessing (on 16 physical cores) like this:

pool = Pool(int(torch.multiprocessing.cpu_count()/2), maxtasksperchild=maxchilds)
download_func = partial(download)
img_dicts = pool.map(download_func, url_dicts)
pool.close()
pool.join()
gc.collect()


def download(url_dict):
try:
    image = io.imread(url_dict["img_url"])
except Exception:
    image = None
gc.collect()

return {"post_id": url_dict["post_id"], "img": image}

The download works fine and increases the speed significantly, except memory usage. During the download, Python allocates ~ 100gb of RAM, although only ~ 2gb are actually used. Each process allocates more than 6 gb of RAM, just for downloading maximum 4 images with ~ 300-700kb each. maxchildspertask were set to 3 to avoid high memory usage, but did not bring significant improvements.

I am running on Windows, Python 3.7, 64 bit

Do you experienced the same or do you have ideas, how i can fix the memory issue?

Edit

I just realized that even I use a download function in an seperate file, which is just imported when in the method it's needed, the imports,... of a service get loaded for each child process, even I use a if __name__ in ('__main__', '__test__', '__routes__', '__shell__', '__runserver__'): guard in my main file application.py. This loading of the whole app is causing the memory issue. How can I prevent, that my subprocesses load the whole app, even if I use a guard in my main file?

Answer 1

Psychic debugging tells me you're allocating some big stuff in your main script without guarding it with an if __name__ == '__main__': guard, e.g. your main script looks like:

# Perform imports (some of which might contain their own definitions of big_stuff)

# Define classes and/or functions

big_stuff = make_big_stuff()

# ... do more things ...

# ... eventually run code that makes Pool ...

rather than:

# Perform cheap imports

# Define classes and/or functions

def main():
    # Perform any imports which might contain definitions of their own big_stuff
    big_stuff = make_big_stuff()

    # ... do more things ...

    # ... eventually run code that makes Pool ...

if __name__ == '__main__':
    main()

When you do the former, every newly spawned worker dutifully recreates big_stuff on launch (even if you don't need it), attempting to emulate forking behavior to ensure all globals in the main script are defined the same in both parent and child. As you've seen, this leads to explosive memory use. The solution is to guard execution of any code that:

1. Consumes excessive resources, and
2. Doesn't produce anything the worker processes need direct access to

with the:

if __name__ == '__main__':

guard, so it only runs in the parent process (the workers are launched with the main script being run with a name like '__mp_main__' or similar, so they won't execute code guarded like that). It means anything the child processes don't have access to due to the guard must be passed if it's needed (so if that 6 GB of data is needed in all processes, you'll need to look into shared memory to avoid duplicating it), but at least you won't be consuming it immediately on launch in cases where the workers don't need it at all.