can't pickle dictionary in django

Question

I have a simple dictionary that i am trying to save to cache and looks like it django is trying to pickle:

  podcasts = []
  for i in items:
        s = re.sub('[\s+]', '', str(i))
        s2 = re.findall(r'<link/>(.*?)<itunes',s)[0]

        item_obj = {}
        item_obj['title'] = title
        item_obj['url'] = s2
        item_obj['created_at'] = created_at
        item_obj['duration'] = duration

        podcasts.append(item_obj)

This has a very simple format that outputs:

[{'title': "Podcast1", 'url': 'https://example.com\\n', 'created_at': 'Thu, 28 Dec 2017', 'duration': '00:30:34'}]

I am running this from a custom management command like this:

python3 manage.py podcast_job

I attempt to save to cache:

    podcasts = get_podcasts()
    print(podcasts)
    cache.set('podcasts', podcasts)

I get the error:

File "podcast_job.py", line 13, in handle
    cache.set('podcasts', podcasts)
  File "python3.6/site-packages/django_redis/cache.py", line 33, in _decorator
    return method(self, *args, **kwargs)
  File "python3.6/site-packages/django_redis/cache.py", line 68, in set
    return self.client.set(*args, **kwargs)
  File "python3.6/site-packages/django_redis/client/default.py", line 109, in set
    nvalue = self.encode(value)
  File "python3.6/site-packages/django_redis/client/default.py", line 329, in encode
    value = self._serializer.dumps(value)
  File "python3.6/site-packages/django_redis/serializers/pickle.py", line 33, in dumps
    return pickle.dumps(value, self._pickle_version)

RecursionError: maximum recursion depth exceeded while calling a Python object

If I try to save with a string I get no error and it saves fine:

   cache.set('podcasts', str(podcasts))

How can I save the list of dictionaries and not get the error above?

Edit your question to show how `item_obj` is initially created. — John Gordon, Jan 05 '18 at 20:11
This is the second time you’ve asked about this error but I still don’t think you’ve included enough information to reproduce the error. Please show the full management command that isn’t working. — Alasdair, Jan 05 '18 at 22:01
@Alasdair I guess in python3 cache needs to be stored as json instead of a dictionary? This solved my problem. But you are the Python master, you let me know. — Atma, Jan 05 '18 at 22:05

score 2 · Answer 1 · answered Jan 05 '18 at 20:14

2

If you are using datetime objects for created_at and duration, make sure you render them to strings.

answered Jan 05 '18 at 20:14

Dash Winterson

1,197
8
19

that sounds interesting. Can you elaborate? basically, every object shouldn't be pickled if not necessary. – Jean-François Fabre Jan 05 '18 at 20:15
A python datetime object contains things that cannot be cast to string or are invokable (such as class methods). It seems like it may be invoking a method such as .now() that returns another datetime object and doing so over and over until it reaches recursion depth. – Dash Winterson Jan 11 '18 at 03:47

score 1 · Answer 2 · answered Jan 05 '18 at 20:14

1

Pickle does not deal well with functions.

Check out this answer for some insight: https://stackoverflow.com/a/1253813/4225229

You could serialize the result of the function (try json.dumps()) and cache that.

answered Jan 05 '18 at 20:14

JacobIRR

8,545
8
39
68

score 0 · Answer 3 · answered Jan 05 '18 at 22:04

0

I converted the dictionary with json as Jacob suggested like this:

cache.set('podcasts', json.dumps(podcasts))

answered Jan 05 '18 at 22:04

Atma

29,141
56
198
299

can't pickle dictionary in django

3 Answers3