For couple of days now, I am stuck on my machine learning project. I have a python script that should transform the data for model training by a second script. In the first script is a list of arrays that I would like to dump to the disk, the second unpickle it.
I tried using pickle
several times, but every time the script attempts pickling, I get memory error:
Traceback (most recent call last):
File "Prepare_Input.py", line 354, in <module>
pickle.dump(Total_Velocity_Change, file)
MemoryError
And sometime, this script is forced to stop running with a Killed
message.
I also tried using hickle
however, the script keeps running for long time with hickle
dumping huge file of nearly 10GB (du -sh myfile.hkl
) when left overnight. I am certain there no way the array size can exceeds 1.5GB at most. I can also dump the array to the console (print
). Using hickle
, I had to killed the process to stop the script running.
I also tried all the answers here, unfortunately, none worked for me.
Does anyone have an idea how I can safely dump my file to disk for later loading?
Using dill I get the following errors:
Traceback (most recent call last):
File "Prepare_Input.py", line 356, in <module>
dill.dump(Total_Velocity_Change, fp)
File "/home/akil/Desktop/tmd/venv/lib/python3.7/site-packages/dill/_dill.py", line 259, in dump
Pickler(file, protocol, **_kwds).dump(obj)
File "/home/akil/Desktop/tmd/venv/lib/python3.7/site-packages/dill/_dill.py", line 445, in dump
StockPickler.dump(self, obj)
File "/home/akil/anaconda3/lib/python3.7/pickle.py", line 437, in dump
self.save(obj)
File "/home/akil/anaconda3/lib/python3.7/pickle.py", line 504, in save
f(self, obj) # Call unbound method with explicit self
File "/home/akil/anaconda3/lib/python3.7/pickle.py", line 819, in save_list
self._batch_appends(obj)
File "/home/akil/anaconda3/lib/python3.7/pickle.py", line 843, in _batch_appends
save(x)
File "/home/akil/anaconda3/lib/python3.7/pickle.py", line 504, in save
f(self, obj) # Call unbound method with explicit self
File "/home/akil/anaconda3/lib/python3.7/pickle.py", line 819, in save_list
self._batch_appends(obj)
File "/home/akil/anaconda3/lib/python3.7/pickle.py", line 843, in _batch_appends
save(x)
File "/home/akil/anaconda3/lib/python3.7/pickle.py", line 504, in save
f(self, obj) # Call unbound method with explicit self
File "/home/akil/anaconda3/lib/python3.7/pickle.py", line 819, in save_list
self._batch_appends(obj)
File "/home/akil/anaconda3/lib/python3.7/pickle.py", line 843, in _batch_appends
save(x)
File "/home/akil/anaconda3/lib/python3.7/pickle.py", line 549, in save
self.save_reduce(obj=obj, *rv)
File "/home/akil/anaconda3/lib/python3.7/pickle.py", line 638, in save_reduce
save(args)
File "/home/akil/anaconda3/lib/python3.7/pickle.py", line 504, in save
f(self, obj) # Call unbound method with explicit self
File "/home/akil/anaconda3/lib/python3.7/pickle.py", line 774, in save_tuple
save(element)
File "/home/akil/anaconda3/lib/python3.7/pickle.py", line 504, in save
f(self, obj) # Call unbound method with explicit self
File "/home/akil/anaconda3/lib/python3.7/pickle.py", line 735, in save_bytes
self.memoize(obj)
File "/home/akil/anaconda3/lib/python3.7/pickle.py", line 461, in memoize
self.memo[id(obj)] = idx, obj
MemoryError