I am trying to use PyDREAM to sample a likelihood that has a number of dynamically constructed elements, i.e., class factories. The class factories are pretty necessary, so making the likelihood function easier to pickle is not really an option. This doesn't have much to do with PyDREAM, as there are a number of "out of the box" samplers that use pickling of some sort for multiprocessing. I assume this is pretty standard since the pickling happens in the multiprocessing module. I'd like to figure out if there is a way to make them work with my code. I was really excited to find cloudpickle which can successfully pickle my likelihood function.
I recently forked PyDREAM and tried this monkey patch. I have successfully patched cloudpickle in, but multiprocessing is trying to call a method called register, which does not seem to exist in cloudpickle. I know nothing about the inner workings of these picklers. There are other methods that start with "register" in cloudpickle, but they don't seem quite right.
~/anaconda3/envs/dream/lib/python3.9/multiprocessing/sharedctypes.py in rebuild_ctype(type_, wrapper, length)
136 if length is not None:
137 type_ = type_ * length
--> 138 _ForkingPickler.register(type_, reduce_ctype)
139 buf = wrapper.create_memoryview()
140 obj = type_.from_buffer(buf)
AttributeError: type object 'CloudPickler' has no attribute 'register'
Also, I've tried using dill to serialize the likelihood with no luck. It would be awesome if multiprocess allowed the use of cloudpickle, and there is an issue on the multiprocess GitHub page about this, but it doesn't seem to be a feature that is being actively worked on.
