multiprocessing a function with parameters that are iterated through

Question

I'm trying to improve the speed of my program and I decided to use multiprocessing!

the problem is I can't seem to find any way to use the pool function (i think this is what i need) to use my function

here is the code that i am dealing with:

def dataLoading(output):
    name = ""
    link = ""
    upCheck = ""
    isSuccess = ""
    for i in os.listdir():
        with open(i) as currentFile:
            data = json.loads(currentFile.read())
            try:
                name = data["name"]
                link = data["link"]
                upCheck = data["upCheck"]
                isSuccess = data["isSuccess"]
            except:
                print("error in loading data from config: improper naming or formating used")
            output[name] = [link, upCheck, isSuccess]

#working
def userCheck(link, user, isSuccess):
    link = link.replace("<USERNAME>", user)
    isSuccess = isSuccess.replace("<USERNAME>", user)
    html = requests.get(link, headers=headers)
    page_source = html.text
    count = page_source.count(isSuccess)
    if count > 0:
        return True
    else:
        return False

I have a parent function to run these two together but I don't think i need to show the whole thing, just the part that gets the data iteratively:

    for i in configData:
        data = configData[i]
        link = data[0]
        print(link)
        upCheck = data[1] #just for future use
        isSuccess = data[2]
        if userCheck(link, username, isSuccess) == True:
            good.append(i)

you can see how I enter all of the data in there, how would I be able to use multiprocessing to do this when I am iterating through the dictionary to collect multiple parameters?

Answer 1

I like to use mp.Pool().map. I think it is easiest and most straight forward and handles most multiprocessing cases. So how does map work? For starts, we have to keep in mind that mp creates workers, each worker receives a copy of the namespace (ya the whole thing), then each worker works on what they are assigned and returns. Hence, doing something like "updating a global variable" while they work, doesn't work; since they are each going to receive a copy of the global variable and none of the workers are going to be communicating. (If you want communicating workers you need to use mp.Queue's and such, it gets complicated). Anyway, here is using map:

from multiprocessing import Pool

t = 'abcd'
def func(s):
    return t[int(s)]

results = Pool().map(func,range(4))

Each worker received a copy of t, func, and the portion of range(4) they were assigned. They are then automatically tracked and everything is cleaned up in the end by Pool.

Something like your dataLoading won't work very well, we need to modify it. I also cleaned the code a little.

def loadfromfile(file):
    data  = json.loads(open(file).read())
    items = [data.get(k,"") for k in ['name','link','upCheck','isSuccess']]
    return items[0],items[1:]

output = dict(Pool().map(loadfromfile,os.listdir()))