Questions tagged [cloudpickle]
12 questions
4
votes
1 answer
cloudpickle of object from imported class versus class defined in the same module as where pickling occurs
I noticed that the pickle file resulting from cloudpickle.dump(obj) is different depending on whether the class of obj (call it SubClass, a subclass of BaseClass) is imported or defined in the same module as where the cloudpickling occurs.
In…
morfys
- 2,195
- 3
- 28
- 35
2
votes
0 answers
Monkey patching cloudpickle as multiprocessing pickler
I am trying to use PyDREAM to sample a likelihood that has a number of dynamically constructed elements, i.e., class factories. The class factories are pretty necessary, so making the likelihood function easier to pickle is not really an option.…
BanjoKayaker
- 21
- 2
2
votes
1 answer
Different cloudpickle everytime the program is run
Consider the following python code:
import cloudpickle
class Foo:
def __init__(self, num):
self.num = num
def outer(num):
return Foo(num)
print(cloudpickle.dumps(outer))
This produces a different pickle everytime you run the…
skgbanga
- 2,477
- 2
- 20
- 29
2
votes
0 answers
How can I serialize a class that contains methods that call other classes in python?
I am trying to parallelize some code that creates an instance of a Predictor class. This class contains
many methods and attributes and looks something like this:
class Predictor:
def__init__(...):
def load_models(...):
def…
michaelarman
- 51
- 4
1
vote
1 answer
Class variable is not shared over all instances after pickling and unpickling
I have a class ConfProvider that contains a class variable which holds all config that cannot be pickled. My other classes inherit from this class and they use the _conf class variable when they call the non-pickable objects. The idea is then to…
Lennart
- 383
- 4
- 16
1
vote
0 answers
pickle/cloudpickle Python 3.7 in colab installed but not found
On my laptop I use pickle abundantly with the routine
def read_pickle(source_file):
with open(source_file, "rb") as fp:
data = pickle.load(fp)
fp.close()
return data
When using that as
data = read_pickle(datafile)
it throws…
user9165100
- 371
- 3
- 11
0
votes
1 answer
cloudpickle: unexpectly import my local module when it is not necessary
In file ai2_kit/domain.py
def fun(ctx):
def in_add(a, b):
print (a+b)
ctx.executor.run_python_fn(in_add)(1, 2) # this pass
ctx.executor.run_python_fn(out_add)(1, 2) # this failed, the error is: ModuleNotFoundError: No module…
link89
- 1,064
- 10
- 13
0
votes
0 answers
Cannot pickle dateparser using cloudpickle
I'm using the dateparser library to parse some strings and return potential dates. I need to use cloudpickle for distributed use but am receiving an error:
import dateparser
class DateParser:
def __init__(self,
threshold:…
Drivebyluna
- 344
- 2
- 14
0
votes
0 answers
Pickle ModuleNotFoundError
I have a python object foo that I want to serialize so I run:
with open('foo.pkl', 'wb') as file:
pickle.dump(foo, file)
I then submit the serialized object to a microservice in another virtual environment, now the problem is, foo depends on a…
João Areias
- 1,192
- 11
- 41
0
votes
1 answer
How come cythonize code works when running on Spark standalone and YARN client deployment mode, but not on YARN cluster deployment?
I have a Python library that I have cythonized using this approach. On all of my Spark cluster nodes, I have installed the whl file as follows.
pip install myapi-0.0.1-cp38-cp38-linux_x86_64.whl
When I submit a job to Spark standalone as follows,…
Jane Wayne
- 8,205
- 17
- 75
- 120
0
votes
1 answer
TypeError: can't pickle Struct objects
I am the maintainer of the python package Construct and I seek help in making this library picklable. Someone came to me and asked for it to be cloudpickle-able. Unfortunately the classes I have are not pickle-able nor cloudpickle-able nor…
ArekBulski
- 4,520
- 4
- 39
- 61
-1
votes
1 answer
Python3 C-libs: How do I serialize a class that depends on C .dlls for use in a bare instance?
For example:
// file foo
def my_func():
print("foo!")
// file bar
class Bar:
def __init__(self):
pass
def hello():
import foo.my_func
my_func()
import dill
dill.dump(Bar(), open("bar.dpkl",mode='wb+'))
$: python3 -m…
Chris
- 28,822
- 27
- 83
- 158