kombu.exceptions.EncodeError: User is not JSON serializable

Question

I have django 1.11.5 app with celery 4.1.0 and I recived all the time:

kombu.exceptions.EncodeError: <User: testuser> is not JSON serializable

my settings.py:

CELERY_BROKER_URL = 'amqp://localhost'
CELERY_RESULT_BACKEND = 'amqp://localhost'
CELERY_ACCEPT_CONTENT = ['application/json']
CELERY_RESULT_SERIALIZER = 'json'
CELERY_TASK_SERIALIZER = 'json'
CELERY_TIMEZONE = 'Asia/Makassar'
CELERY_BEAT_SCHEDULE = {}

tasks.py

from __future__ import absolute_import, unicode_literals
from celery import task
from django.contrib.auth.models import User


@task(serializer='json')
def task_number_one():
    user = User.objects.create(username="testuser", email="test@test.com", password="pass")
    return user

I call task in the view:

def form_valid(self, form):
    form.instance.user = self.request.user
    task_number_one.delay()
    return super().form_valid(form)

This error also occurs when passing an object to a task from a view. See MrName's answer below. — MarMat, May 25 '21 at 09:58

score 37 · Answer 1 · answered Mar 19 '18 at 23:43

37

This is because you are using the JSON serializer for task serialization (as indicated by the setting CELERY_TASK_SERIALIZER = 'json'), but you are trying to return a model instance (which cannot be serialized into JSON).

You have two options:

1) Don't pass the instance, pass the primary key of the instance and then look up the object inside your task.

2) Use the pickle task serializer instead. This will allow you to pass objects as arguments to your tasks and return them, but comes with it's own security concerns.

answered Mar 19 '18 at 23:43

MrName

2,363
17
31

thanks a lot for answer. In point 1, you mean something like this (with pk): `get_object_or_404(User, pk=self.kwargs['pk'])`? – user9357277 Mar 20 '18 at 00:18
@MrName from **Celery 4.0** onwards they are using `json` as the default serializer, see [this change log](http://docs.celeryproject.org/en/latest/whatsnew-4.0.html#json-is-now-the-default-serializer) – JPG Mar 20 '18 at 01:56
Sorry, I misspoke. As @JerinPeterGeorge pointed out, the error is coming from your RETURN value as opposed to the arguments being passed into your task. His answer is completely correct UNLESS you (for some reason) need to actually return the user object as the result (this is probably not the case). in this case, you would need to set `CELERY_RESULT_SERIALIZER = 'json'` – MrName Mar 20 '18 at 16:04
This does not solve my problem. I have the exact same issue. – waqasgard Apr 03 '19 at 05:39
1

The answer is correct for an almost similar situation(when trying to pass objects or class instances as arguments in a task) but not for the question asked. For the question, @JPG is right – Mitch Nov 08 '19 at 08:41

score 20 · Accepted Answer · answered Mar 20 '18 at 02:10

The error is because of Celery expecting a JSON data from your task function while you returned a User instance.

How to solve this ?
You are not using that return data anywhere, so you don't have to return it. That is you can remove return user from the task function.
Or,
return a Json data from the task function will solve this issue as well

Solution 1

@task(serializer='json')
def task_number_one():
    user = User.objects.create(username="testuser", email="test@test.com", password="pass")

Solution 2

@task(serializer='json')

def task_number_one():
    user = User.objects.create(username="testuser", email="test@test.com", password="pass")
    # return some json data instead of `USER` instance
    return {"status": True}  # Change is here

score 0 · Answer 3 · answered Feb 22 '23 at 06:46

My celery task:

response = {}
...
except HTTPError as e:
        response.update(
            {
                'status': False,
                'code': e.status_code,
                'error': e.body,
            },
        )
    ...
return response

I had EncodeError(TypeError('Object of type bytes is not JSON serializable') and kombu.exceptions.EncodeError although the response is a dict that shouldn't be a problem when JSON encoding.

It turned out that e.body is of type bytes. I changed to e.body.decode('utf-8') and the problem disappeared.

score 0 · Answer 4 · answered Jul 02 '23 at 23:07

Another answer not related to this question but also useful

if you pass obj to task, may got the same error info, then you can:

@shared_task(serializer="pickle")
def email_send_task(msg: EmailMultiAlternatives):
    try:
        msg.send()
    except (smtplib.SMTPException, TimeoutError) as e:
        return f"Email failed with {e}"

score 0 · Answer 5 · answered Jul 26 '23 at 08:36

For the following configuration:

Django==3.0.11
redis==2.10.6
celery==4.0.0
kombu==4.0.0

I updated the following configuration on the settings.py file and it worked.

CELERY_SETTINGS = {
    'CELERY_TIMEZONE': TIME_ZONE,
    'CELERY_ENABLE_UTC': True,
    'CELERY_RESULT_BACKEND': REDIS_URL,
    'CELERY_SEND_TASK_SENT_EVENT': True,
    'CELERY_TASK_SERIALIZER': 'pickle',
    'CELERY_RESULT_SERIALIZER': 'pickle',
    'CELERY_ACCEPT_CONTENT': ['pickle', 'json'],
}

Don't forget to update the actual value of TIME_ZONE and REDIS_URL

The reason could be the Celery 4 version uses JSON as the serializer by default, while the celery3 version uses Pickle by default.

The old Django may not be expecting the JSON format from the tasks. So, If you are using the very old Django version, this could help you.

kombu.exceptions.EncodeError: User is not JSON serializable

5 Answers5

Linked