How do you get amazon SQS to work with Django celery

Question

I am trying to get ques to work with Amazon SQS but tasks aren't run. However My tasks run out of the box with rabbitmq

here are the links i have tried (among others)

https://www.caktusgroup.com/blog/2011/12/19/using-django-and-celery-amazon-sqs/

They seem to be quite dated. on my cues I do not get any messages received when i log into the amazon console but i see the celery logs have activity showing they are communicating with Amazon

    DEBUG b'<?xml version="1.0"?><ReceiveMessageResponse xmlns="http://queue.amazonaws.com/doc/2012-11-05/"><ReceiveMessageResult/><ResponseMetadata><RequestId>f8d10c14-99f7-520b-a58d-5d2cc203ad8b</RequestId></ResponseMetadata></ReceiveMessageResponse>'
    Mar 7 08:56:52 ip-172-31-41-253 e-celery-worker.log DEBUG Method: GET
    Mar 7 08:56:52 ip-172-31-41-253 e-celery-worker.log DEBUG Path: /967610578225/transfer_files
    Mar 7 08:56:52 ip-172-31-41-253 e-celery-worker.log DEBUG Data: 
    Mar 7 08:56:52 ip-172-31-41-253 e-celery-worker.log DEBUG Headers: {}
    Mar 7 08:56:52 ip-172-31-41-253 e-celery-worker.log DEBUG Host: eu-west-1.queue.amazonaws.com
    Mar 7 08:56:52 ip-172-31-41-253 e-celery-worker.log DEBUG Port: 443
    Mar 7 08:56:52 ip-172-31-41-253 e-celery-worker.log DEBUG Params: {'Version': '2012-11-05', 'Action': 'ReceiveMessage', 'WaitTimeSeconds': 0, 'MaxNumberOfMessages': 4}
    Mar 7 08:56:52 ip-172-31-41-253 e-celery-worker.log DEBUG Token: None
    Mar 7 08:56:52 ip-172-31-41-253 e-celery-worker.log DEBUG CanonicalRequest:
    Mar 7 08:56:52 ip-172-31-41-253 e-celery-worker.log  GET
    Mar 7 08:56:52 ip-172-31-41-253 e-celery-worker.log  /967610578225/transfer_files
    Mar 7 08:56:52 ip-172-31-41-253 e-celery-worker.log  Action=ReceiveMessage&MaxNumberOfMessages=4&Version=2012-11-05&WaitTimeSeconds=0
    Mar 7 08:56:52 ip-172-31-41-253 e-celery-worker.log  host:eu-west-1.queue.amazonaws.com
    Mar 7 08:56:52 ip-172-31-41-253 e-celery-worker.log  x-amz-date:20170307T065652Z
    Mar 7 08:56:52 ip-172-31-41-253 e-celery-worker.log
, 'Authorization': 'AWS4-HMAC-SHA256 Credential=xxxxxx/20170307/eu-west-1/sqs/aws4_request,SignedHeaders=host;x-amz-date,Signature=xxxxxx', 'User-Agent': 'Boto/2.45.0 Python/3.4.3 Linux/4.1.17-22.30.amzn1.x86_64', 'X-Amz-Date': '20170307T065652Z'}
    Mar 7 08:56:52 ip-172-31-41-253 eright-celery-worker.log DEBUG Response headers: [('Server', 'Server'), ('Date', 'Tue, 07 Mar 2017 06:56:52 GMT'), ('Content-Type', 'text/xml'), ('Content-Length', '240'), ('Connection', 'keep-alive'), ('x-amzn-RequestId', 'bbad9a79-f65e-568d-aaeb-cb41adaa390d')]

my settings are

BROKER_TRANSPORT = 'sqs'
BROKER_TRANSPORT_OPTIONS = {
    'region': 'eu-west-1',
    'polling_interval': 2,
    'visibility_timeout': 3600,
}
BROKER_USER = 'xxx'
BROKER_PASSWORD = 'xxxxx/xxxx'

CELERY_IMPORTS = (
    'apps.files.tasks',
)

CELERY_QUEUES = {
    'default': {
        'binding_key': 'default'
    },

    # Files
    'files_copy_paste': {
        'binding_key': 'files.copy.paste'
    },
}

Any help would be tremendously appreciated

score 3 · Accepted Answer · answered Mar 07 '17 at 10:37

3

I have Django + Celery working with SQS with the following settings:

import urllib.parse

# Defined in environment settings
AWS_ACCESS_KEY_ID = os.environ["AWS_ACCESS_KEY_ID"] 
AWS_SECRET_ACCESS_KEY = os.environ["AWS_SECRET_ACCESS_KEY"]

CELERY_BROKER_URL = 'sqs://{0}:{1}@'.format(
    urllib.parse.quote(AWS_ACCESS_KEY_ID, safe=''),
    urllib.parse.quote(AWS_SECRET_ACCESS_KEY, safe='')
)

CELERY_BROKER_TRANSPORT_OPTIONS = {
    'region': 'sqs.eu-west-1',
    'queue_name_prefix': 'celery-'
}

Have you specified your AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY?

I also found that I had to put sqs. before my region name.

answered Mar 07 '17 at 10:37

Isaac Jordan

130
8

Hi @Isaac, the problem is i need AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY set for S3 so i use the s3 users access. and for the broker i asumed BROKER_USER = 'xxx' BROKER_PASSWORD = 'xxxxx/xxxx' is where i put my access key and id. Your point of putting the sqs in front of the region is the only thing i haven't tried, i will get right on it and let you know if it works – Tawanda M Mar 07 '17 at 13:04
@TawandaMinya Hey, did it work? :) If so, please accept my answer. – Isaac Jordan Mar 07 '17 at 19:52
Hi, Have not gotten a chance to test yet. But when i do will def – Tawanda M Mar 09 '17 at 09:23
@TawandaMinya Just another check up on whether my answer helped :) If so, please accept it! – Isaac Jordan May 07 '17 at 11:24
It did not solve my particular problem, but since you are the only one that answered I have accepted. Thank you for the assist – Tawanda M May 08 '17 at 12:11
@TawandaM Is it posible to know how did you solve your problem? – Marcos Aguayo Feb 26 '20 at 19:21

How do you get amazon SQS to work with Django celery

1 Answers1