5

I have a celery worker running on Elastic Beanstalk that polls a SQS queue, gets messages (containing S3 file names), downloads those files from S3 and processes them. My worker is scheduled to run at every 15 seconds but due to some reason the memory usage keeps on increasing with time.

This is the code I'm using to access SQS

def get_messages_from_sqs(queue_url, queue_region="us-west-2", number_of_messages=1):
    client = boto3.client('sqs', region_name=queue_region)
    sqs_response = client.receive_message(QueueUrl=queue_url, MaxNumberOfMessages=number_of_messages)
    messages = sqs_response.get("Messages", [])
    cleaned_messages = []
    for message in messages:
        body = json.loads(message["Body"])
        data = body["Records"][0]
        data["receipt_handle"] = message["ReceiptHandle"]
        cleaned_messages.append(data)
    return cleaned_messages

def download_file_from_s3(bucket_name, filename):
    s3_client = boto3.client('s3')
    s3_client.download_file(bucket_name, filename, '/tmp/{}'.format(filename))

Do we need to close client connection in boto3 after we're done with the process ? If so, how can we do it ? Memory Monitor Graph

Aakash
  • 97
  • 1
  • 9
  • If the system didn't free up memory soon enough , try this. https://stackoverflow.com/questions/1316767/how-can-i-explicitly-free-memory-in-python – mootmoot May 15 '18 at 09:17
  • which celery version are you using? – Arghya Saha Aug 20 '18 at 10:13
  • **celery 4.1.0** – Aakash Aug 21 '18 at 06:33
  • If you suspect a memory leak, the way to identify it is to create a memory dump at some point when memory usage is high and try to understand what type of objects occupy the memory. This may give then an idea about which part of the implementation leaks the memory. You can use `Guppy` and `Heapy` for this. – Tim Aug 24 '18 at 10:31
  • on completely another side, try to reuse `client` instances, instead of creating new instance with every call to `get_messages_from_sqs` – Tim Aug 24 '18 at 10:32
  • which version of boto3 are you using? – James Lim Aug 24 '18 at 15:13

1 Answers1

2

I have run into similar issues using Celery in production, completely unrelated to Boto. Although I do not have an explanation for the memory leak (this would take some serious code spelunking and profiling), I can offer a potential workaround if your goal is just to not run out of memory.

Setting max tasks per child should allow you to constantly reclaim the memory as it is released by the killed process.

MrName
  • 2,363
  • 17
  • 31
  • The bug I'm looking for is boto3 related for sure – JBernardo Aug 23 '18 at 19:16
  • I definitely believe that, I just wasn't sure if your question related specifically to trying to diagnose the memory leak, or how to get celery to move past it and keep things running. – MrName Aug 24 '18 at 18:32