0

I'm trying to update a piece of code, and the initializer value/function used for multiprocessing.Pool() is causing an AttributeError when I run it. I've had a look through the documentation and a stack of relevant(ish) Google results, but I can't find anything explaining what the initializer can and can't be or any significant detail on how to use it. How do I find more information on this?

EDIT: The code up trying to update is this snippet from Real Python's concurrency article.

import requests
import multiprocessing
import time

session = None


def set_global_session():
    global session
    if not session:
        session = requests.Session()


def download_site(url):
    with session.get(url) as response:
        name = multiprocessing.current_process().name
        print(f"{name}:Read {len(response.content)} from {url}")


def download_all_sites(sites):
    with multiprocessing.Pool(initializer=set_global_session) as pool:
        pool.map(download_site, sites)


if __name__ == "__main__":
    sites = [
        "https://www.jython.org",
        "http://olympus.realpython.org/dice",
    ] * 80
    start_time = time.time()
    download_all_sites(sites)
    duration = time.time() - start_time
    print(f"Downloaded {len(sites)} in {duration} seconds")

When I run that it's just repeatedly outputting "AttributeError: Can't get attribute 'set_global_session' on ".

mangoUnchained
  • 39
  • 5
  • 1
    How are you using it? Show a [mcve]. Afaik, it's just a function that's called. – Carcigenicate Jan 07 '20 at 22:54
  • Does this answer your question? [how to use initializer to set up my multiprocess pool?](https://stackoverflow.com/questions/10117073/how-to-use-initializer-to-set-up-my-multiprocess-pool) – Jimmar Jan 07 '20 at 22:59
  • I've updated the question. @Jimmar, for global variables, that discussion seems to be recommending basically what's in the code I'm using. – mangoUnchained Jan 07 '20 at 23:09
  • That is weird, I just tested your code and it seems to run properly, mind sharing which version of python are you using and how are you running the code ? (I used `Python 3.7.4`) – Jimmar Jan 07 '20 at 23:22
  • code works without error on Linux Mint 19.2 (based on Ubuntu 18.04), Python 3.7.6 – furas Jan 07 '20 at 23:28
  • As an aside, is this the best way to share a Session? – AMC Jan 07 '20 at 23:35
  • I'm using a Jupyter Notebook from the Anaconda distribution. So, the answer is some combination of these: python 3.7.4, ipython 7.8.0, jupyter 1.0.0, jupyter_client 5.3.3, jupyter_console 6.0.0, jupyter_core 4.5.0, jupyterlab 1.1.4, jupyterlab_server 1.0.6. – mangoUnchained Jan 07 '20 at 23:58
  • I've just checked, and it works from the command line. Does multiprocessing.Pool() just not work in a notebook? – mangoUnchained Jan 08 '20 at 00:22

0 Answers0