4

I am trying to run a static method on one of my classes in my Django application. I am using the multiprocessing module to make this task a little faster since the method will be iterating over a large amount of objects in my database. It works fine when I run it locally, but when I run it in production I get this pickling error...

Quick Note: I using python3.6 locally, and python3.4 in production. Would this affect the pickling of my objects ?

Traceback (most recent call last):
  File "<console>", line 1, in <module>
  File "/home/socialauto/social-auto-web/vehicle/models.py", line 193, in image_existence_check
    p.map(Vehicle.check_image, enumerate(vehicles))
  File "/usr/lib/python3.4/multiprocessing/pool.py", line 260, in map
    return self._map_async(func, iterable, mapstar, chunksize).get()
  File "/usr/lib/python3.4/multiprocessing/pool.py", line 599, in get
    raise self._value
  File "/usr/lib/python3.4/multiprocessing/pool.py", line 383, in _handle_tasks
    put(task)
  File "/usr/lib/python3.4/multiprocessing/connection.py", line 206, in send
    self._send_bytes(ForkingPickler.dumps(obj))
  File "/usr/lib/python3.4/multiprocessing/reduction.py", line 50, in dumps
    cls(buf, protocol).dump(obj)
_pickle.PicklingError: Can't pickle <function Vehicle.check_image at 0x7f49974d5268>: attribute lookup check_image on vehicle.models failed

Por Que

Model method:

    @staticmethod
    def check_image(veh):
        index = veh[0]
        print(index)

        veh = veh[1]

        try:
            images = veh.images.all()

            if images.count() == 1:
                image = images[0]

                response = requests.get(image.image_url)

                if response.status_code == 200:
                    veh.has_image = True
                else:
                    veh.has_image = False
            elif images.count() > 1:
                has_image = True

                for img in images:
                    response = requests.get(img.image_url)

                    if response != 200:
                        has_image = False

                veh.has_image = has_image

            else:

                veh.has_image = False

            veh.save()

        except Exception as e:
            logging.error(e)
            pass

    @staticmethod
    def image_existence_check():
        from time import time
        from multiprocessing.pool import Pool

        ts = time()
        vehicles = Vehicle.objects.all()[:100]

        # map(lambda (i, x): {'name': x, 'rank': i}, enumerate(ranked_users))

        with Pool(10) as p:
            print('Beginning')
            p.map(Vehicle.check_image, enumerate(vehicles))

        print('Took {}s'.format(time() - ts))
TJB
  • 3,706
  • 9
  • 51
  • 102
  • 1
    Possible duplicate of [Can't pickle when using multiprocessing Pool.map()](https://stackoverflow.com/questions/1816958/cant-pickle-type-instancemethod-when-using-multiprocessing-pool-map) – Evya Dec 17 '17 at 00:34

0 Answers0