Stack Overflow
Stack Overflow

Questions tagged [kombu]

Kombu is an AMQP messaging framework for Python.

Kombu is an AMQP messaging framework for Python. Kombu provides a high-level API for messaging using the AMQP standard.

Allows application authors to support several message server solutions by using pluggable transports.

  • Supports multiple transports(Redis, Beanstalk, CouchDB, and MongoDB. SQLAlchemy and Django ORM)
  • Supports automatic encoding, serialization and compression of message payloads.
184 questions
58
votes
7 answers

Celery with Amazon SQS

I want to use Amazon SQS as broker backed of Celery. There’s the SQS transport implementation for Kombu, which Celery depends on. However there is not enough documentation for using it, so I cannot find how to configure SQS on Celery. Is there…
minhee
  • 5,688
  • 5
  • 43
  • 81
40
votes
4 answers

In-Memory broker for celery unit tests

I have a REST API written in Django, with and endpoint that queues a celery task when posting to it. The response contains the task id which I'd like to use to test that the task is created and get the result. So, I'd like to do something like: def…
Facundo Olano
  • 2,492
  • 2
  • 26
  • 32
29
votes
1 answer

Can anyone please tell me what are the differences between pika and kombu messaging library in python?

I want to use messaging library in my application to interact with rabbitmq. Can anyone please explain the differences between pika and kombu library?
Nitesh Singh Rajput
  • 607
  • 1
  • 7
  • 24
22
votes
3 answers

Rabbitmq error: [Errno 10054] An existing connection was forcibly closed by the remote host

I am using Kombu in Python to consume a durable RabbitMQ queue. There is only one consumer consuming the queue in Windows. This consumer produces the below error: Traceback (most recent call last): File ".\consumer_windows.py", line 66, in…
Greg
  • 8,175
  • 16
  • 72
  • 125
14
votes
1 answer

What are the django-celery (djcelery) tables for?

When I run syncdb, I notice a lot of tables created like: djcelery_crontabschedule ... djcelery_taskstate django-kombu is providing the transport, so it can't be related to the actual queue. Even when I run tasks, I still see nothing populated in…
jgoldberg
  • 455
  • 5
  • 11
11
votes
2 answers

Celery beat not starting EOFError('Ran out of input')

Everything worked perfectly fine until: celery beat v3.1.18 (Cipater) is starting. __ - ... __ - _ Configuration -> . broker -> amqp://user:**@staging-api.user-app.com:5672// . loader -> celery.loaders.app.AppLoader .…
Benjamin Toueg
  • 10,511
  • 7
  • 48
  • 79
10
votes
1 answer

Celery upgrade (3.1->4.1) - Connection reset by peer

We are working with celery at the last year, with ~15 workers, each one defined with concurrency between 1-4. Recently we upgraded our celery from v3.1 to v4.1 Now we are having the following errors in each one of the workers logs, any ideas what…
ItayB
  • 10,377
  • 9
  • 50
  • 77
10
votes
2 answers

Differentiate celery, kombu, PyAMQP and RabbitMQ/ironMQ

I want to upload images to S3 server, but before uploading I want to generate thumbnails of 3 different sizes, and I want it to be done out of request/response cycle hence I am using celery. I have read the docs, here is what I have understood.…
user2116985
  • 103
  • 1
  • 4
9
votes
1 answer

Celery not accepting pickle even after allowing it

I'm trying to write a celery application that passes numpy arrays (or any arbitrary objects) to the workers. As far as I can tell, this requires serialization to occur via pickle (NB: I'm aware of the security implications but this isn't a concern…
janfreyberg
  • 358
  • 2
  • 11
9
votes
1 answer

Django & Celery — Routing problems

I'm using Django and Celery and I'm trying to setup routing to multiple queues. When I specify a task's routing_key and exchange (either in the task decorator or using apply_async()), the task isn't added to the broker (which is Kombu connecting to…
Michael Waterfall
  • 20,497
  • 27
  • 111
  • 168
8
votes
2 answers

Celery: Accessing the Broker Connection Pool

I'm using Celery with an AMQP broker to call tasks, but the response needs to be passed back with a different queue architecture than Celery uses, so I want to pass the messages back using Kombu only. I've been able to do this, but I'm creating a…
wolverdude
  • 1,583
  • 1
  • 13
  • 20
8
votes
3 answers

Celery Shutting down

After upgrading from celery 2.4.5 i have started having celery randomly shutdown. I am using celery 3.0.12, boto 2.6 and amazon sqs and django 1.2.7 all this on a centOS machine (pip freeze dump at the bottom) i am running service celerybeat…
yossi
  • 12,945
  • 28
  • 84
  • 110
7
votes
1 answer

Consuming a rabbitmq message queue with multiple threads (Python Kombu)

I have a single RabbitMQ exchange with a single queue. I wish to create a daemon that runs multiple threads and works through this queue as quickly as possible. The "work" involves communicating with external services, so there will be a fair…
Steve Mayne
  • 22,285
  • 4
  • 49
  • 49
7
votes
1 answer

Airflow Running task from UI, KeyError: No such transport

airflow cfg settings related to celery are: broker_url = 'amqp://guest:guest@rabbitmq_server:8080' celery_result_backend = db+postgresql://developer:password@postgres_server:5432/db_name The airflow webserver runs ok, but while running a task from…
Javed
  • 5,904
  • 4
  • 46
  • 71
7
votes
2 answers

Celery, kombu and django - import error

I am running an application with django, and I wanted to use celery to make some scheduled tasks. According to the oficial docs, in my settings.py file I set the broker transport BROKER_URL = 'django://' and added kombu.transport.django to…
Gabriel Ilharco
  • 1,649
  • 1
  • 21
  • 34
1
2 3
…
12 13