1

I just recently started to play with celery. in my project, I want to use Python celery workers and handle different tasks. However, some tasks are from a Java program. of course I can choose to execute a small python script from java with parameters--but that may actually cause a lot of script invoking and therefore slow.

I found some solutions/tutorials online about Java + RabbitMQ + Celery. Seems the json is not complicated. (Tutorial and one SO question and another SO question). However, our dev stack does not contain RabbitMQ and therefore we want to just make use of MongoDB as the broker. I followed the code here for creating a simple task and add it to the queue. Then I checked my MongoDB and I find this:

{ "_id" : ObjectId("578f4db162091b0ca50b6e5d"), "queue" : "celery", "priority" : 9, "payload" : "{\"body\": \"eyJleHBpcmVzIjogbnVsbCwgInV0YyI6IHRydWUsICJhcmdzIjogWzIsIDJdLCAiY2hvcmQiOiBudWxsLCAiY2FsbGJhY2tzIjogbnVsbCwgImVycmJhY2tzIjogbnVsbCwgInRhc2tzZXQiOiBudWxsLCAiaWQiOiAiNWVhMTQ2N2YtYzM4Ny00MGE2LTlhY2QtZGNlMjhhYmQyYjJiIiwgInJldHJpZXMiOiAwLCAidGFzayI6ICJ0YXNrcy5hZGQiLCAiZ3JvdXAiOiBudWxsLCAidGltZWxpbWl0IjogW251bGwsIG51bGxdLCAiZXRhIjogbnVsbCwgImt3YXJncyI6IHt9fQ==\", \"headers\": {}, \"content-type\": \"application/json\", \"properties\": {\"priority\": 0, \"body_encoding\": \"base64\", \"correlation_id\": \"5ea1467f-c387-40a6-9acd-dce28abd2b2b\", \"reply_to\": \"85d8b4f0-1490-3212-b4d0-a9b6c69de7b0\", \"delivery_info\": {\"routing_key\": \"celery\", \"exchange\": \"\"}, \"delivery_mode\": 2, \"delivery_tag\": \"a4c2149c-17a6-4666-8ccc-d0e8e1b34a84\"}, \"content-encoding\": \"utf-8\"}" }

And when I decode the payload body as a base64 string, I got this:

{"expires": null, "utc": true, "args": [2, 2], "chord": null, "callbacks": null, "errbacks": null, "taskset": null, "id": "5ea1467f-c387-40a6-9acd-dce28abd2b2b", "retries": 0, "task": "tasks.add", "group": null, "timelimit": [null, null], "eta": null, "kwargs": {}}

Which is effectively some JSON string according to the message protocol. So do have to construct a JSON string this way and encode in base64, and construct another JSON string to wrap it. And then save it to the database? Any easier solution? Or am I understanding any part wrongly?

So I want the java program to add some tasks to the queue.

For id in body, is it just a random UUID? What about reply_to?

Community
  • 1
  • 1
Junchao Gu
  • 1,815
  • 6
  • 27
  • 43
  • Why are you trying to build message yourself? You could configure mongodb as backend just like rabbitmq and let celery handle the rest. – atv Jul 20 '16 at 16:05
  • See this page http://docs.celeryproject.org/en/latest/getting-started/brokers/mongodb.html . – atv Jul 20 '16 at 16:06
  • But java program should add tasks to the queue. That is my concern. – Junchao Gu Jul 21 '16 at 00:42
  • Oh in that case, you have not other option but to construct the message in java code and put in the db. Another 'hacky' way is to use flower api. You can connect flower with mongodb backend and then use rest calls to queue tasks . If you are okay with this approach I can put details in the answer. – atv Jul 21 '16 at 03:44
  • You can go ahead and post your answers :) – Junchao Gu Jul 21 '16 at 05:20
  • Unfortunately flower supports only rabbit and redis broker. So this solution will not work. https://github.com/mher/flower/blob/master/flower/utils/broker.py – atv Jul 21 '16 at 05:38

0 Answers0