Configure Celery with JSON Serializer (Python + node.js)

Question

I'm running a Celery worker in Python with the celery module v3.1.25, and a Celery client in node.js with the node-celery npm package v0.2.7 (not the latest).

The Python Celery worker works fine when sending a job using a Python Celery client.

Problem: When using a node-celery client to send a task to the Celery backend, we get an error in the JS console:

(STDERR) Celery should be configured with json serializer

Python Celery worker is configured with:

app = Celery('tasks', 
    broker='amqp://test:test@192.168.1.26:5672//',
    backend='amqp://',
    task_serializer='json',
    include=['proj.tasks'])

node-celery client is configured with:

var celery = require('node-celery')
var client = celery.createClient({
    CELERY_BROKER_URL: 'amqp://test:test@192.168.1.26:5672//',
    CELERY_RESULT_BACKEND: 'amqp',
    CELERY_TASK_SERIALIZER: "json"
});

client.on('connect', function() {
    console.log('connected');

    client.call('proj.tasks.getPriceEstimates', [start_latitude, start_longitude],
        function(result) {
            console.log('result: ', result);
            client.end();
        })
});

Is this a problem with the configuration on the Python Celery worker? Did we miss out on a configuration parameter which can change the return serialization format to json?

Update

Updated with result_serializers and accept_content parameters as suggested by ChillarAnand

from __future__ import absolute_import, unicode_literals
from celery import Celery

app = Celery('tasks', 
    broker='amqp://test:test@192.168.1.26:5672//',
    backend='amqp://',
    task_serializer='json',
    result_serializer='json',
    accept_content=['application/json'],
    include=['proj.tasks'])

But node.js Celery client still thinks that its not in json, throwing the same error message.

It gives that error because the results were in the form of 'application/x-python-serialize'.

Checked this to be the case, as RabbitMQ management console shows the results to be content_type: application/x-python-serialize

This forum post says that it is because the tasks were created before the configs were loaded.

Here are how my files are like:

proj/celery.py

from __future__ import absolute_import, unicode_literals
from celery import Celery

app = Celery('tasks', 
    broker='amqp://test:test@192.168.1.26:5672//',
    backend='amqp://',
    task_serializer='json',
    result_serializer='json',
    accept_content=['application/json'],
    include=['proj.tasks'])

proj/tasks.py

from __future__ import absolute_import, unicode_literals
from .celery import app

@app.task
def myTask():
    ...
    return ...

Is there a better way to structure the code to ensure that the configs are loaded before the tasks?

Did you manage to solve your problem? – leoschet Jun 28 '18 at 00:21 — leoschet, Jun 28 '18 at 00:21

score 1 · Answer 1 · answered May 02 '17 at 05:18

1

When configuring a serializer, you should specify content type, task serializer and result serializer as well.

app = Celery(
    broker='amqp://guest@localhost//',
    backend='amqp://',
    include=['proj.tasks'],

    task_serializer='json',
    result_serializer='json',
    accept_content = ['application/json'],
)

answered May 02 '17 at 05:18

Chillar Anand

27,936
9
119
136

I made the suggested changes for my Python Celery worker. but the node client is still giving the same error when receiving the results. I believe it gives that error because the [results was in the form of `'application/x-python-serialize'`](https://github.com/mher/node-celery/blob/master/celery.js?MobileOptOut=1#L337-L338). Any other setting to make to switch it to `json`? – Nyxynyx May 02 '17 at 14:03
For celery_node app, did you set `CELERY_ACCEPT_CONTENT=['json'],CELERY_TASK_SERIALIZER='json', CELERY_RESULT_SERIALIZER='json'`? – Chillar Anand May 02 '17 at 18:21

Configure Celery with JSON Serializer (Python + node.js)

Update

1 Answers1