12

I need to submit a number of jobs to a Laravel queue to process some uploaded CSV files. These jobs could be finished in one second if the files are small, or a few seconds if they're bigger, or possibly up to a minute if the CSV files are very big. And I can't tell in advance how big the files will be.

When the user goes to the "results" page, I need to display the results - but only if the queue has finished the jobs. If the queue is still processing, I need to display a "try again later" message.

So - is there a way to check, from a controller, whether the queue has finished?

I'm currently using Laravel 5.1 but would happily upgrade if that helps. And I'm currently using the database queue driver. Ideally I'd love to find a general technique that works for all queue drivers, but if the only way to do it is to check a database table then I guess that's what I have to do.

Thanks!

Brendan White
  • 435
  • 2
  • 8
  • 17
  • ...so this isn't actually an answer, but a workaround. When I submit the job, I also add a record in a table called "import_jobs", with a "processed" field set to "N". When I've finished processing the job in the queue, I update the record in the "import_jobs" table to have "processed" = "Y". And before showing the results, I first search for records in the "import_jobs" table with "processed" set to "N". If there are any, I list them and say "Come back later". If there are none, I go ahead and display the results. Not beautiful, but I haven't thought of a better workaround. :-( – Brendan White Aug 19 '16 at 03:47
  • Why do you deem this "not beautiful"? This is a pretty valid solution. In order to be able to query the status of a process, you need to have that information saved, updated, and ready for query somewhere. Database is totally ok for that purpose. – Sergey Neskhodovskiy Apr 01 '17 at 17:33
  • It means I have to have a whole new table to separately track what is happening in a queue, and I have to keep that separate table up-to-date manually. But the queue already has some method to know whehter a job is running, or queued, or finished, or crashed or whatever. It seems to me that it would be cleaner to simply ask the queue somehow. But if it's not possible, then that's OK. I can just use the separate table. – Brendan White Apr 13 '17 at 08:39
  • @BrendanWhite I'm interested in doing something similar as your solution, how did you keep track of specific jobs using this other table? I'm using a `report_name` and the `processed` column you mention. The issue I'm facing now is that there can be reports by the same name, so I'm having trouble differentiating between these in the table to change the `processed` column. What do you suggest? – Nancy Jul 28 '20 at 18:48
  • Hi Nancy, in the end we changed direction and never implemented this. But you could create the job in the `import_jobs` table and then save the ID of that new record, and check against the ID instead of the `report_name`. – Brendan White Jul 29 '20 at 22:14

3 Answers3

10

I know this is a year old, but why not create a new queue per upload with a unique key based on that request.

$job = (new ProcessCSVJob($data))->onQueue($uniqueQueueName);

You can then simply either do a count in the database on the queue name field if you want a DB only solution.

To work across all queue types you can use the Queue size method to return the queue size.

$queue = App::make('queue.connection');
$size = $queue->size($uniqueQueueName);

This is in Laravel 5.4. Not sure how backwards compatible this is.

Robert Norman
  • 191
  • 2
  • 5
  • 6
    This works great in L5.6. An alternative way to get the size is through the `Queue` facade: `$size = \Queue::size($uniqueQueueName);`. I've noticed some queue drivers may raise an exception if the named queue does not exist. I'm not sure if that is a bug in the drivers though, but it's something to consider. Using the facade may be more robust, in case the IoC container name ('queue.connection') changes internally. – Jason Mar 08 '18 at 16:48
2

I expect if I was trying to do this today, I'd use a Laravel Job Event to update a status field on the database record, to log when the job has started and when it's finished.

Then I could see whether a record has been fully processed or not by just looking at the status field on the record itself.

Brendan White
  • 435
  • 2
  • 8
  • 17
0

since version 8, laravel provides the concept of batches, which you can use to monitor progress of jobs.

woens
  • 1,058
  • 9
  • 20