24

I have set up queues in Laravel for my processing scripts. I am using beanstalkd and supervisord. There are 6 different tubes for different types of processing.

The issue is that for each tube, artisan is constantly spawning workers every second. The worker code seems to sleep for 1 second and then the worker thread uses 7-15% cpu, multiply this by 6 tubes... and I would like to have multiple workers per tube.. my cpu is being eaten up.

I tried changing the 1 second sleep to 10 seconds. This helps but there is still a huge cpu spike every 10 seconds when the workers wake back up. I am not even processing anything at this time because the queues are completely empty, it is simply the workers looking for something to do.

I also tested to see the cpu usage of laravel when I refreshed the page in a brower and that was hovering around 10%.. I am on a low end rackspace instance right now so that could explain it but still... it seems like the workers spin up a laravel instance every time they wake up.

Is there no way to solve this? Do I just have to put a lot of money into a more expensive server just to be able to listen to see if a job is ready?

EDIT:

Found a solution... it was to NOT use the artisan queue:listener or queue:work I looked into the queue code and there doesn't seem to be a way around this issue, it requires laravel to load every time a worker checks for more work to do.

Instead I wrote my own listener using pheanstalk. I am still using laravel to push things into the queue, then my custom listener is parsing the queue data and then triggering an artisan command to run.

Now my cpu usage for my listeners is under %0, the only time my cpu shoots up now is when it actually finds work to do and then triggers the command, I am fine with that.

bsparacino
  • 333
  • 1
  • 3
  • 8

3 Answers3

14

The problem of high CPU is caused because the worker loads the complete framework everytime it checks for a job in the queue. In laravel 4.2, you can use php artisan queue:work --daemon. This will load the framework once and the checking/processing of jobs happen inside a while loop, which lets CPU breathe easy. You can find more about daemon worker in the official documentation: http://laravel.com/docs/queues#daemon-queue-worker.

However, this benefit comes with a drawback - you need special care when deploying the code and you have to take care of the database connections. Usually, long running database connections are disconnected.

MohitMamoria
  • 558
  • 5
  • 8
  • 1
    thanks for the update. good to know that they finally support this. I have been using beanstalkd for over a year now and it's great, I did have to take care of database connections as you mentioned. – bsparacino Aug 29 '14 at 00:37
  • Yes, they officially recommend to run `DB::reconnect()` before processing every job, to make sure that the connection is available. The `reconnect()` method disconnects the already established connection and connects it again. – MohitMamoria Aug 29 '14 at 09:26
  • Im guessing that if you use docker and place the `php artisan queue:work --daemon` command as the CMD of the container it will restart automatically on a failed DB connection. – AndrewMcLagan Oct 06 '16 at 22:27
5

I had the same issue.

But I found another solution. I used the artisan worker as is, but I modified the 'watch' time. By default(from laravel) this time is hardcoded to zero, I've changed this value to 600 (seconds). See the file: 'vendor/laravel/framework/src/Illuminate/Queue/BeanstalkdQueue.php' and in function 'public function pop($queue = null)'

So now the work is also listening to the queue for 10 minutes. When it does not have a job, it exits, and supervisor is restarting it. When it receives a job, it executes it after that it exists, and supervisor is restarting it.

==> No polling anymore!

notes:

  • it does not work for iron.io queue's or others.
  • it might not work when you want that 1 worker accept jobs from more than 1 queue.
karelv
  • 756
  • 9
  • 20
  • 1
    The solution I posted to my first posted has been working great for me so far. I thought about modifying the file you said but I would rather not modify the vendor/core files, especially since I don't commit those directories, it is part of my build process to dynamically generate those directories on the live server. Thanks for the help though! – bsparacino Jul 31 '13 at 17:34
  • 1
    I found this to be the best solution in regards to performance. The new commit in Laravel that adds a sleep parameter it somewhat problematic since while the process is sleeping it will not look for any new jobs. Also I noticed that it still has some issues with CPU usage. The above solution reduced my cpu down to 1% and it also continuously checks for jobs. – greatwitenorth Oct 15 '13 at 19:25
  • 1
    @greatwitenorth, could you do something to improve my reputation here on SO? thanks. – karelv Nov 24 '13 at 12:04
  • Yes, the watch time is the issue. Even with the beanstalkd queue, Laravel does not passively connect to a queue and listen for a message; instead it polls the queues, over and over, with no delay and a zero watch time. It's crazy IMO. It does this *just in case* there is one listener that needs to watch multiple queues, so it polls each queue in a round-robin fashion. However, if your listener only has one queue, it does not fall back to a more efficient non-polling listen method. No, it keeps on polling, eating up CPU. It really needs to be fixed; been like this for too long. It's now 2015. – Jason Oct 22 '15 at 22:07
2

According to this commit, you can now set a new option to queue:listen "--sleep={int}" which will let you fine tune how much time to wait before polling for new jobs.
Also, default has been set to 3 instead of 1.

younes0
  • 2,288
  • 1
  • 25
  • 34
  • The documentation is not even slightly clear about what this "sleep" actually does. Does it sleep once at the start, ad then starts polling (rapidly)? It is supposed to sleep for that long between each poll? Does it sleep only if it did not find a message when polling? In my 5.1 trials, I can tell you it is none of the above, so what its purpose is, escapes me completely. – Jason Oct 22 '15 at 22:12
  • You answered your own question: it sleeps between each poll and only if no message is found as said in http://laravel.com/docs/5.1/queues. It's very clear to me :) did you use sync as queue mode in your trials? – younes0 Oct 23 '15 at 06:23
  • "the number of seconds to wait before polling for new jobs" - maybe it is the wording that is ambiguous to me. The number of seconds to wait *between each check while polling*, would be clearer. "before polling" is the time between starting the process and starting the polling loop, and that *appears* to be what has been implemented. Anyway, I have not seen the sleep option working on L5.1 - polling runs full speed with no sleep period being applied. – Jason Oct 26 '15 at 17:02
  • if you think that the docs are wrong (which I don't doubt they may be), feel free to open an issue. Did you run the artisan command in the appropriate env? what about dumping queue options to check if you're in sync mode or not ? see also: https://github.com/laravel/framework/issues/6206 – younes0 Oct 26 '15 at 17:32