Error while getting list from another process

Question

I wanted to write small program that would symulate for me lottery winning chances. After that i wanted to make it a bit faster by implemening multiprocessing like this

But two weird behaviors started

import random as r
from multiprocessing.pool import ThreadPool



# winnerSequence = []
# mCombinations = []

howManyLists = 5
howManyTry = 1000000
combinations = 720/10068347520


possbilesNumConstantsConstant = []
for x in range(1, 50):
    possbilesNumConstantsConstant.append(x)

def getTicket():
    possbilesNumConstants = list(possbilesNumConstantsConstant)

    toReturn = []
    possiblesNum = list(possbilesNumConstants)

    for x in range(6):
        choice = r.choice(possiblesNum)
        toReturn.append(choice)
        possiblesNum.remove(choice)
    toReturn.sort()
    return toReturn


def sliceRange(rangeNum,num):
    """returns list of smaller ranges"""
    toReturn = []
    rest = rangeNum%num
    print(rest)
    toSlice = rangeNum - rest
    print(toSlice)
    n = toSlice/num
    print(n)
    for x in range(num):
        toReturn.append((int(n*x),int(n*(x+1)-1)))
    print(toReturn,"<---range")
    return toReturn

def Job(tupleRange):
    """Job returns list of tickets """
    toReturn = list()
    print(tupleRange,"Start")
    for x in range(int(tupleRange[0]),int(tupleRange[1])):
        toReturn.append(getTicket())
    print(tupleRange,"End")
    return toReturn


result = list()

First one when i add Job(tupleRange) to pool it looks like job is done in main thread before another job is added to pool

def start():
    """this fun() starts program"""

    #create pool of threads
    pool = ThreadPool(processes = howManyLists)

    #create list of tuples with smaller piece of range
    lista = sliceRange(howManyTry,howManyLists)

    #create list for storing job objects
    jobList = list()
    for tupleRange in lista:
        #add job to pool
        jobToList = pool.apply_async(Job(tupleRange))
        #add retured object to list for future callback
        jobList.append(jobToList)
        print('Adding to pool',tupleRange)

    #for all jobs in list get returned tickes
    for job in jobList:
        #print(job.get())
        result.extend(job.get())

if __name__ == '__main__':
    start()

Consol output

[(0, 199999), (200000, 399999), (400000, 599999), (600000, 799999), (800000, 999999)] <---range
(0, 199999) Start
(0, 199999) End
Adding to pool (0, 199999)
(200000, 399999) Start
(200000, 399999) End
Adding to pool (200000, 399999)
(400000, 599999) Start
(400000, 599999) End

and second one when i want to get data from thread i got this exception on this line

for job in jobList:
        #print(job.get())
        result.extend(job.get())  #<---- this line



File "C:/Users/CrazyUrusai/PycharmProjects/TestLotka/main/kopia.py", line 79, in start
    result.extend(job.get())
  File "C:\Users\CrazyUrusai\AppData\Local\Programs\Python\Python36\lib\multiprocessing\pool.py", line 644, in get
    raise self._value
  File "C:\Users\CrazyUrusai\AppData\Local\Programs\Python\Python36\lib\multiprocessing\pool.py", line 119, in worker
    result = (True, func(*args, **kwds))
TypeError: 'list' object is not callable

Can sombody explain this to me?(i am new to multiprocessing)

score 1 · Accepted Answer · answered Aug 27 '18 at 09:53

The problem is here:

jobToList = pool.apply_async(Job(tupleRange))

Job(tupleRange) executes first, then apply_async gets some returned value, list type (as Job returns list). There are two problems here: this code is synchronous and async_apply gets list instead of job it expects. So it try to execute given list as a job but fails.

That's a signature of pool.apply_async:

def apply_async(self, func, args=(), kwds={}, callback=None,
            error_callback=None):
     ...

So, you should send func and arguments args to this function separately, and shouldn't execute the function before you will send it to the pool.

I fix this line and your code have worked for me:

jobToList = pool.apply_async(Job, (tupleRange, ))

Or, with explicitly named args,

jobToList = pool.apply_async(func=Job, args=(tupleRange, ))

Don't forget to wrap function arguments in tuple or so.

Error while getting list from another process

1 Answers1