1

Short

Is there an equivalent to the multiprocessing.Pool()'s map function for non-picklable objects in python 3? Or do I misunderstand this error? :

TypeError: can't pickle memoryview objects

Long

I have a large whoosh index (text indexing and searching library) which sometimes takes time to query.

So in order to speed up computations I split it into 5 different indexes so that I can query them with 5 different processes.

I do not want to re-open the index every time so I want to feed my search function with an opened index as argument. This object is not picklable so I can't use multiprocessing.Pool()'s map function to run 5 processes querying the 5 different (opened) indexes.

I therefore tried multiprocessing.dummy.Pool() to do the job with threads but it does not seem to really speed up computations.

  • Am I missing something here and there is an easier way to speed up such computations?
  • In Python 3 is there a way to use multiple processes for such non-picklable objects?
  • Maybe I misunderstand this error : TypeError: can't pickle memoryview objects
ted
  • 13,596
  • 9
  • 65
  • 107
  • Relevant: [how-can-i-make-this-un-pickleable-object-pickleable](https://stackoverflow.com/questions/30699391/python-how-can-i-make-this-un-pickleable-object-pickleable) – stovfl Aug 01 '17 at 18:18

0 Answers0