7

So I am using django with celery. rabbitmq is the broker. redis is the cache. mysql is the db. (everything in localhost)

  1. I am using python2.7 and using virtualenv based virtual environment
  2. I start the redis server (local) at default port

  3. In a new terminal, I run 

    python manage.py runserver

  4. In a new terminal I start celery like so

    celery -A ds_django worker -B -l warning

And this procedure used to work. But now when I run the celery command, I get the following 

    [2016-07-12 09:15:20,113: CRITICAL/MainProcess] Frequent restarts detected: RestartFreqExceeded('5 in 1s',)
    Traceback (most recent call last):
        File "/Users/user/Desktop/ds-django/ds_django/newDs/lib/python2.7/site- packages/celery/worker/consumer.py", line 285, in start
            self._restart_state.step()
        File "/Users/user/Desktop/ds-django/ds_django/newDs/lib/python2.7/site-packages/billiard/common.py", line 130, in step
            raise self.RestartFreqExceeded("%r in %rs" % (R, self.maxT))
     RestartFreqExceeded: 5 in 1s

I have absolutely no clue why this is happening or how to fix this. I have searched Google and StackOverFlow over and over again, but no use.

One reason why I think it might have happened is, sometime back I had run 4 celery workers at-a-time and then cold-shut them all down. Maybe because of that it is saying restart frequency exceeded, but not sure. And no idea how to fix this. Probably some config file setting, but which setting and where also no clue.

If any other info is required, I will provide. Any help is appreciated. Thank you.

TheRajVJain
  • 390
  • 5
  • 15

1 Answers1

6

To answer your question, Your RabbitMQ is down due to that consumer started to reconncet and it went to contineous loop and it created an exception RestartFreqExceeded. Please try starting your RabbitMQ server and run your celery.

backtrack
  • 7,996
  • 5
  • 52
  • 99
  • I have been trying that @Backtrack. Same error still. I uninstalled and installed rabbitmq (using homebrew). No use. Still the same error – TheRajVJain Jul 12 '16 at 04:48
  • @RajJain, Are you able to access RabbitMQ ? – backtrack Jul 12 '16 at 04:50
  • @RajJain, Are you giving localhost to connect of remote ip in the consumer ? – backtrack Jul 12 '16 at 04:51
  • I get this when I run `rabbitmqctl status` [{pid,3641}, {running_applications,[{os_mon,"CPO CXC 138 46","2.4"}, {ssl,"Erlang/OTP SSL application","7.3"}, {mnesia,"MNESIA CXC 138 12","4.13.3"}, {amqp_client,"RabbitMQ AMQP Client","3.6.1"}, {public_key,"Public key infrastructure","1.1.1"}, {asn1,"The Erlang ASN1 compiler version 4.0.2", "4.0.2"}, {syntax_tools,"Syntax tools","1.7"}, and so on @Backtrack – TheRajVJain Jul 12 '16 at 04:58
  • How are you connecting from consumer ? Are you using localhost or remote IP ? – backtrack Jul 12 '16 at 05:00
  • i am using localhost – TheRajVJain Jul 12 '16 at 05:02
  • @RajJain, Are you running the celery task from same machine ? – backtrack Jul 12 '16 at 05:10
  • yes @Backtrack. everything is local and at localhost – TheRajVJain Jul 12 '16 at 05:12
  • update: I run `brew services start rabbitmq`, it says already started. when I run `rabbitmqadmin list queues` it says connection refused. when I go to management console, it says unable to connect to it – TheRajVJain Jul 12 '16 at 05:24
  • `rabbitmq-server` ERROR: node with name "rabbit" already running on "localhost" and then http://localhost:15672/#/queues, it says "undefined: There is no template at js/tmpl/login.ejs" again – TheRajVJain Jul 12 '16 at 05:25
  • @RajJain Which means you are still having RabbitMQ connection issue. Which OS are you suing ? – backtrack Jul 12 '16 at 05:30
  • Mac OS X @Backtrack – TheRajVJain Jul 12 '16 at 06:04
  • @RajJain - Can you run this : rabbitmqctl status and check the status. Please follow this :https://www.rabbitmq.com/install-standalone-mac.html – backtrack Jul 12 '16 at 06:09
  • I did that. i reinstalled again. And then when i run `rabbitmq-server` stops at `Starting broker...` for almost forever @Backtrack – TheRajVJain Jul 12 '16 at 06:21
  • Please check the above link – backtrack Jul 12 '16 at 08:41
  • The answer is correct. there was some problem with my rabbitmq. I uninstalled rabbitmq, deleted ALL files regarding it (with the help of my senior), then reinstalled and it worked :) but the above link was not so useful :) @Backtrack – TheRajVJain Jul 13 '16 at 10:15
  • @RajJain, Consider upvoting :) glad it could help – backtrack Jul 13 '16 at 10:23
  • i did! but apparently due to less than 15 rep, it won't show in public lol. when my rep becomes more than 15, it will show. So fingers-croosed :) – TheRajVJain Jul 13 '16 at 11:12
  • @RajJain, :). lol. Plz follow this 2 links, We are using celery in PRD and the follwoing links will be useful. https://denibertovic.com/posts/celery-best-practices/ https://blog.balthazar-rouberol.com/celery-best-practices – backtrack Jul 13 '16 at 11:17
  • Nice!! Thanks @Backtrack – TheRajVJain Jul 14 '16 at 07:05
  • I'm seeing this error and my RabbitMQ server is just fine. – Cerin Sep 03 '19 at 17:54
  • @Cerin - When you say the Rabbitmq is running fine - How did you verify that ? Are you running both Celery and Rabbit on the same server ? – backtrack Sep 04 '19 at 06:23
  • I saw a recent down vote - dear down voters please leave your message. I think i have helped OP to resolve this issue. If my answer did not help you please create one. I am happy to help you. – backtrack Sep 05 '19 at 13:13