1

I have the following celery setup in production

  • Using RabbitMQ as the broker
  • I am running multiple instances of the celery using ECS Fargate
  • Logs are sent to CloudWatch using default awslogs driver
  • Result backend - MongoDB

The issue I am facing is this. A lot of my tasks are not showing logs on cloudwatch.

I just see this log

Task task_name[3d56f396-4530-4471-b37c-9ad2985621dd] received

But I do not see the actual logs for the execution of this task. Nor do I see the logs for completion - for example something like this is nowhere in the logs to be found

Task task_name[3d56f396-4530-4471-b37c-9ad2985621dd] succeeded

This does not happen all the time. It happens intermittently but consistently. I see that a lot of tasks are printing the logs.

I can see that result backend has the task results and I know that the task has executed but the logs for the task are completely missing. It is not specific to some task_name.

On my local setup, I have not been able to isolate the issue

I am not sure if this is a celery logging issue or awslogs issue. What can I do to troubleshoot this?

** UPDATE **

Found the root cause - it was that I had some code in the codebase that was removing handlers from the root logger. Leaving this question on stack overflow in case someone else faces this issue

Log all requests from the python-requests module

Ranjith Ramachandra
  • 10,399
  • 14
  • 59
  • 96
  • 1
    Could it be something as simple as container got removed before the logs were sent to the CloudWatch? – DejanLekic Jul 12 '22 at 09:17
  • The containers are alive and I can see that there are more successful logs for tasks in the container after this event – Ranjith Ramachandra Jul 12 '22 at 09:29
  • 1
    I was merely thinking about the case when container gets terminated. – DejanLekic Jul 12 '22 at 09:34
  • I am so out of ideas - I may even setup kibana just to check if this is an aws issue or the celery task issue. – Ranjith Ramachandra Jul 12 '22 at 16:29
  • @DejanLekic I finally found the root cause - It was not anything to do with celery. What was going on was that I had a http debugger as per this thread and once it executed, handlers from root logger got removed. I copied the debug statement from this thread https://stackoverflow.com/questions/16337511/log-all-requests-from-the-python-requests-module Turns out its dangerous to copy the code from stackoverflow :D – Ranjith Ramachandra Jul 21 '22 at 09:53

0 Answers0