OS error load averages unobtainable greenlet celery stackoverflow

Question

I am frequently running into this error. Error Image

This is happening when I am running celery worker with Gevent Thread. This error starts coming frequently after its first incidence.

I am using Python 2.7 Celery 4.0.2 OS Ubuntu 14.04

Answer 1

This is basically OS error stating that it is running out of file descriptors. Celery is running 2000 tasks in parallel which need 4000 task descriptors. Default limit of OS is (ulimit - n) Soft 1024 Hard 65000 Updating these limits resolved the issue. To update these limits follow steps on this link How to increase Neo4j's maximum file open limit (ulimit) in Ubuntu?