0

i want to use apply_async from multiprocessing python library (python 2.7 in Ubuntu 16.04) in a class, my code for example is :

class Try_multiprocess():
def multyprocese_chunks(self,chunks):
    pool = Pool(processes=2)
    site_id=564
    site_st= 564
    for chunk_ix, chunk in enumerate(chunks):
        pool.apply_async(self.execute_chunk, args=(chunk, chunk_ix, site_id, site_st,))

    print "{} wait for join".format(datetime.datetime.now())
    pool.close()
    pool.join()
    print "{} after for join".format(datetime.datetime.now())


def execute_chunk(self, chunk, chunk_ix, site_id, site_st):
    print "{} execute_chunk chunk : {} ".format(datetime.datetime.now(), chunk_ix)

and this does not work (nothing prints and no error) i read somewhere that instance method can't be serialized but is there any workaround ? maybe with static/class method's ? or any other way rather than extract all methods fro m the class ?

Omer Anisfeld
  • 1,236
  • 12
  • 28

1 Answers1

1

according to Python multiprocessing PicklingError: Can't pickle <type 'function'> , the answer is no : multiprocessing can only transfer Python objects to worker processes which can be pickled. If you cannot reorganize your code as described by unutbu, https://thelaziestprogrammer.com/python/a-multiprocessing-pool-pickle also suggest that there is no way of multiprocessing no pickle objects

Omer Anisfeld
  • 1,236
  • 12
  • 28