Python celery logging rotation

Question

I have a celery application and that executes a task with concurrency = 1. I added the log using following code, it is working fine however when it tries to rotate the log i am facing exception,

below is my code,

app = Celery('src.tasks',
             broker='amqp://user:pass@server:5672//',
             )

app.config_from_object('src.settings')

# Optional configuration, see the application user guide.
app.conf.update(
    CELERY_ACKS_LATE=True,
    CELERY_PREFETCH_MULTIPLIER=1,
    CELERYBEAT_MAX_LOOP_INTERVAL=600,
    CELERY_ROUTES={'myqueue': {'queue': 'myqueue'}},
    CELERY_TASK_SERIALIZER='pickle',
    CELERY_RESULT_SERIALIZER='pickle',
    CELERY_ACCEPT_CONTENT=['pickle', 'json'],
    result_expires=3600,
)


LOGGER = app.conf.LOGGER

and my setting is,

LOG_FILENAME="log.log"
LOGGER = logging.getLogger(__name__)
LOGGER.setLevel(logging.DEBUG)
handler = logging.handlers.TimedRotatingFileHandler(LOG_FILENAME,
                                               maxBytes=100,
                                               backupCount=5,
                                             )
formatter = logging.Formatter(LOG_FORMAT)
handler.setFormatter(formatter)
LOGGER.addHandler(handler)

I am executing the code like below

>> celery -A src.tasks worker -E -Q AWS_jobclassification --concurrency=1

the error

[2020-12-18 06:40:01,305: WARNING/Worker-1] File "d:\install\miniconda\envs\py27\lib\logging\handlers.py", line 350, in doRollover
[2020-12-18 06:40:01,305: WARNING/Worker-1] os.rename(self.baseFilename, dfn)
[2020-12-18 06:40:01,305: WARNING/Worker-1] WindowsError: [Error 32] The process cannot access the file because it is being used by another process
[2020-12-18 06:40:01,305: WARNING/Worker-1] Logged from file tasks.py, line 183

What is wrong in my code ? and how can i use log rotate in celery task ? Thank you.

i am using Windows server

I suggest you simply use [logrotate](https://linux.die.net/man/8/logrotate) - that is what I do. — DejanLekic, Dec 18 '20 at 13:16

score 1 · Answer 1 · answered Dec 18 '20 at 12:19

1

I do not have any knowledge in Celery. However, from what I see, you are using the logging module, which does not support multiprocessing out of the box. This is explained here.

From that answer, each process will have a file handle, so when one of the processes tries to modify the file (change its name to rotate the log), you will get that error (the other processes still have the file handle opened).

It would be useful to see where in your code you are using the LOGGER. Also, from the documentation, I see Celery provides a logging configuration, which (at least from the code in the post) doesn't seem like you are using it. Maybe that configuration helps solve the concurrency issues you are having.

answered Dec 18 '20 at 12:19

Miguel Alorda

623
5
13

1

Actually if you look at the [Celery code](https://docs.celeryproject.org/en/stable/_modules/celery/app/log.html) you will see that it's setting up a multiprocessing logger. – Tomáš Linhart Dec 18 '20 at 13:19
@TomášLinhart Yes, but in the code shown on this question, the `celery.app.log.Logging` class doesn't seem to be used to configure logging. It seems like the `logging` module is being used directly (which does not support multiprocessing out of the box). – Miguel Alorda Dec 18 '20 at 13:37

Python celery logging rotation

1 Answers1