I am trying to run a static method on one of my classes in my Django application. I am using the multiprocessing module to make this task a little faster since the method will be iterating over a large amount of objects in my database. It works fine when I run it locally, but when I run it in production I get this pickling error...
Quick Note: I using python3.6 locally, and python3.4 in production. Would this affect the pickling of my objects ?
Traceback (most recent call last):
File "<console>", line 1, in <module>
File "/home/socialauto/social-auto-web/vehicle/models.py", line 193, in image_existence_check
p.map(Vehicle.check_image, enumerate(vehicles))
File "/usr/lib/python3.4/multiprocessing/pool.py", line 260, in map
return self._map_async(func, iterable, mapstar, chunksize).get()
File "/usr/lib/python3.4/multiprocessing/pool.py", line 599, in get
raise self._value
File "/usr/lib/python3.4/multiprocessing/pool.py", line 383, in _handle_tasks
put(task)
File "/usr/lib/python3.4/multiprocessing/connection.py", line 206, in send
self._send_bytes(ForkingPickler.dumps(obj))
File "/usr/lib/python3.4/multiprocessing/reduction.py", line 50, in dumps
cls(buf, protocol).dump(obj)
_pickle.PicklingError: Can't pickle <function Vehicle.check_image at 0x7f49974d5268>: attribute lookup check_image on vehicle.models failed
Por Que
Model method:
@staticmethod
def check_image(veh):
index = veh[0]
print(index)
veh = veh[1]
try:
images = veh.images.all()
if images.count() == 1:
image = images[0]
response = requests.get(image.image_url)
if response.status_code == 200:
veh.has_image = True
else:
veh.has_image = False
elif images.count() > 1:
has_image = True
for img in images:
response = requests.get(img.image_url)
if response != 200:
has_image = False
veh.has_image = has_image
else:
veh.has_image = False
veh.save()
except Exception as e:
logging.error(e)
pass
@staticmethod
def image_existence_check():
from time import time
from multiprocessing.pool import Pool
ts = time()
vehicles = Vehicle.objects.all()[:100]
# map(lambda (i, x): {'name': x, 'rank': i}, enumerate(ranked_users))
with Pool(10) as p:
print('Beginning')
p.map(Vehicle.check_image, enumerate(vehicles))
print('Took {}s'.format(time() - ts))