20

The Situation

I'm using Laravel Queues to process large numbers of media files, an individual job is expected to take minutes (lets just say up to an hour).

I am using Supervisor to run my queue, and I am running 20 processes at a time. My supervisor config file looks like this:

[program:duplitron-worker]
process_name=%(program_name)s_%(process_num)02d
command=php /var/www/duplitron/artisan queue:listen database --timeout=0 --memory=500 --tries=1
autostart=true
autorestart=true
user=duplitron
numprocs=20
redirect_stderr=true
stdout_logfile=/var/www/duplitron/storage/logs/duplitron-worker.log

In my duplitron-worker.log I noticed Failed: Illuminate\Queue\CallQueuedHandler@call occurs occasionally and I would like to better understand what exactly is failing. Nothing appears in my laravel.log file (which is where exceptions would normally appear).

The Question

Is there a handy way for me to learn more about what is causing my job to fail?

slifty
  • 13,062
  • 13
  • 71
  • 109
  • 1
    Add `loglevel` under `[supervisord]` in the supervisord.conf. You can start with `debug` (check more here: http://supervisord.org/logging.html). Usually the reason why it fails, because the code you run doesn't end with correct "exit status". You can also see it here: http://stackoverflow.com/questions/28937722/supervisord-exit-status-1-not-expected-running-php-script Last thing, you may need to add `stderr_logfile` in your configs. – Axalix Dec 27 '15 at 23:00

5 Answers5

34

In the newer Laravel versions there's an exception column in the failed_jobs table that has all the info you need. Thanks cdarken and Toskan for pointing this out!

==== OLD METHOD BELOW

Here's what I always do, but first - make sure you have a failed-jobs table! It's well documented, look it up :)

  1. Run the php artisan queue:failed command to list all the failed jobs, and pick the one you're after. Write the ID down.

  2. Then, make sure to stop your queue with supervisorctl stop all duplitron-worker:

  3. Lastly, make sure your .env setting for APP_DEBUG = true.

  4. Then run php artisan queue:retry {step_job_1_id}

  5. Now manually runphp artisan queue:listen --timeout=XXX

If the error is structural (and most are), you should get the failure with debug stack in your log file.

Good luck with debugging :-)

Stan Smulders
  • 6,079
  • 1
  • 27
  • 23
  • This is perfect -- thanks! Now I just need to figure out why it works when I retry it (also... the failed tasks slowly get reprocessed correctly despite the fact that I have --tries set to 1 which is confusing me, but that's a separate question!) – slifty Dec 28 '15 at 01:19
  • I guess one followup question / if you have suggestions you could add to your answer -- are there ways I could get the failure stack in a log file when running in bulk. I suspect part of the issue might be related to constrained resources or possibly timeouts, which aren't happening when only a single thread is running. – slifty Dec 28 '15 at 01:26
  • 2
    Well, like you said - if it runs well solo, it's not script related. You narrowed it down, that's the good news Your best bet then would be to check the system and php logs. Something has to be in there! It's most likely a time-out, or memory shortage :) – Stan Smulders Dec 28 '15 at 08:28
  • 3
    --tries=1 will try to run the job one more time if it fails, I.e a totally of 2 times for each job before it gets sent to the failed table. – dino Dec 29 '15 at 04:09
  • @slifty I'm sure you found the answer by now, but I had a similar issue: My notification was working fine until I queued it. After following the steps above, I checked storage/laravel.log and found that one of the parameters passed into my Notification class, "$custom_message" was coming up as "undefined property". Turns out I needed the statement: "protected $custom_message;" before the "public function __construct...." line. – sersun Nov 09 '16 at 08:11
  • 3
    I don't know if there's an addition in the later versions, but in 5.3 the failed jobs table has a column named exception where you can see what actually failed, no need for the tricks above – cdarken Jul 25 '17 at 20:53
  • what is timeout? milliseconds? seconds? minutes? – Toskan Oct 09 '18 at 23:19
  • Argh! I should not have to dig through the database to find out why a job failed. Isn't there a way to just output the exception on the commandline when running `queue:listen` in the foreground? There's gotta be a better workflow. – Illya Moskvin Apr 27 '20 at 00:33
1

as @cdarken pointed out, the exception can be found in your database table failed_jobs column name exception. Thanks @cdarken I wish his answer would be an answer and not a comment.

Toskan
  • 13,911
  • 14
  • 95
  • 185
0
  • Run these command to create a table failed_jobs in db
php artisan queue:failed-table
php artisan migrate

  • Run queue worker php artisan queue:work --tries=2

  • Check the exception reason in your database table failed_jobs you've just created.

David
  • 826
  • 6
  • 4
0

If you're using database than goto failed_jobs table and look for the exception there.

mepsd
  • 21
  • 1
  • 2
-3

Worked for me, in vendor/laravel/framework/src/Illuminate/Notifications/SendQueuedNotifications.php Just remove "use Illuminate\Queue\SerializesModels;" Line 6 & modify Line 11 to "use Queueable;"

ibrahim.suez
  • 83
  • 1
  • 1
  • 2
    Never edit vendor. It's a bad practice. Try to reach the repository with pull-request if something can be improved. – Ostap Brehin Oct 10 '20 at 18:10