Wrote script in OSX, with multiprocessing. Now windows won't play ball

Question

The program/script I've made works on OSX and linux. It uses selenium to scrape data from some pages, manipulates the data and saves it. In order to be more efficient, I included the multiprocessing pool and manager. I create a pool, for each item in a list, it calles the scrap class, starts a phantomjs instance and scrapes. Since I'm using multiprocessing.pool, and I want a way to pass data between the threads, I read that multiprocessing.manager was the way forward. If I wrote manager = Manager() info = manager.dict([]) it would create a dict that could be accessed by all threads. It all worked perfectly.

My issue is that the client wants to run this on a windows machine (I wrote the entire thing on OSX) I assumed, it would be as simple as installing python, selenium and launching it. I had errors which later lead me to writing if __name__ == '__main__: at the top of my main.py file, and indenting everything to be inside. The issue is, when I have class scrape(): outside of the if statement, it cannot see the global info, since it is declared outside of the scope. If I insert the class scrape(): inside the if __name__ == '__main__': then i get an attribute error saying

AttributeError: 'module' object has no attribute 'scrape'

And if I go back to declaring manager = manager() and info = manager.dict([]) outside of the if __name__ == '__main__' then I get the error in windows about making sure I use if __name__ == '__main__' it doesn't seem like I can win with this project at the moment.

Code Layout...

Imports...
from multiprocessing import Pool
from multiprocessing import Manager

manager = Manager()
info = manager.dict([])
date = str(datetime.date.today())

class do_scrape():
    def __init__():
    def...

def scrape_items():#This contains code which creates a pool and then pool.map(do_scrape, s) s = a list of items

def save_scrape():

def update_price():

def main():

main()

Basically, the scrape_items is called by main, then scrape_items uses pool.map(do_scrape, s) so it calls the do_scrape class and passes the list of items to it one by one. The do_scrape then scrapes a web page based on the item url in "s" then saves that info in the global info which is the multiprocessing.manager dict. The above code does not show any if __name__ == '__main__': statements, it is an outline of how it works on my OSX setup. It runs and completes the task as is. If someone could issue a few pointers, I would appreciate it. Thanks

Answer 1

It would be helpful to see your code, but its sounds like you just need to explicitly pass your shared dict to scrape, like this:

import multiprocessing
from functools import partial

def scrape(info, item):
   # Use info in here

if __name__ == "__main__":
   manager = multiprocessing.Manager()
   info = manager.dict()
   pool = multiprocessing.Pool()
   func = partial(scrape, info) # use a partial to make it easy to pass the dict to pool.map
   items = [1,2,3,4,5] # This would be your actual data
   results = pool.map(func, items)
   #pool.apply_async(scrape, [shared_dict, "abc"]) # In case you're not using map...

Note that you shouldn't put all your code inside the if __name__ == "__main__": guard, just the code that's actually creating processes via multiprocessing, this includes creating the Manager and the Pool.

Any method you want to run in a child process must be declared at the top level of the module, because it has to be importable from __main__ in the child process. When you declared scrape inside the if __name__ ... guard, it could no longer be imported from the __main__ module, so you saw the AttributeError: 'module' object has no attribute 'scrape' error.

Edit:

Taking your example:

import multiprocessing
from functools import partial

date = str(datetime.date.today())

#class do_scrape():
#    def __init__():
#    def...
def do_scrape(info, s):
    # do stuff
    # Also note that do_scrape should probably be a function, not a class

def scrape_items():
    # scrape_items is called by main(), which is protected by a`if __name__ ...` guard 
    # so this is ok.
    manager = multiprocessing.Manager()
    info = manager.dict([])
    pool = multiprocessing.Pool()
    func = partial(do_scrape, info) 
    s = [1,2,3,4,5] # Substitute with the real s
    results = pool.map(func, s)     

def save_scrape():

def update_price():

def main():
    scrape_items()

if __name__ == "__main__": 
    # Note that you can declare manager and info here, instead of in scrape_items, if you wanted
    #manager = multiprocessing.Manager()
    #info = manager.dict([])
    main()

One other important note here is that the first argument to map should be a function, not a class. This is stated in the docs (multiprocessing.map is meant to be equivalent to the built-in map).

Answer 2

Find the starting point of your program, and make sure you wrap only that with your if statement. For example:

Imports...
from multiprocessing import Pool
from multiprocessing import Manager

manager = Manager()
info = manager.dict([])
date = str(datetime.date.today())

class do_scrape():
    def __init__():
    def...

def scrape_items():#This contains code which creates a pool and then pool.map(do_scrape, s) s = a list of items

def save_scrape():

def update_price():

def main():

if __name__ == "__main__":
    main()

Essentially the contents of the if are only executed if you called this file directly when running your python code. If this file/module is included as an import from another file, all attributes will be defined, so you can access various attributes without actually beginning execution of the module.

Read more here: What does if __name__ == "__main__": do?