11

I have created an application to send e-mails to more than one user but I am facing a problem when dealing with a large number of recipients.

The error appears in a failed_jobs table

Illuminate\Queue\MaxAttemptsExceededException: App\Jobs\ESender has been attempted too many times or run too long. The job may have previously timed out. in D:\EmailSender\vendor\laravel\framework\src\Illuminate\Queue\Worker.php:649

and this is payload in failed_jobs table

{"uuid":"ff988083-c1da-4d20-a2e3-c2a10e154c79","timeout":9000,"id":"j2Lz0Ro0bkJpqwxKWTxC3Tiii71iE6Cm","data":{"command":"O:16:\"App\\Jobs\\ESender\":13:{s:7:\"timeout\";i:9000;s:12:\"receiver_obj\";O:45:\"Illuminate\\Contracts\\Database\\ModelIdentifier\":4:{s:5:\"class\";s:12:\"App\\Receiver\";s:2:\"id\";i:6;s:9:\"relations\";a:0:{}s:10:\"connection\";s:5:\"mysql\";}s:16:\"sender_all_hosts\";O:45:\"Illuminate\\Contracts\\Database\\ModelIdentifier\":4:{s:5:\"class\";s:15:\"App\\SenderHosts\";s:2:\"id\";a:4:{i:0;i:1;i:1;i:2;i:2;i:3;i:3;i:4;}s:9:\"relations\";a:0:{}s:10:\"connection\";s:5:\"mysql\";}s:11:\"message_obj\";O:45:\"Illuminate\\Contracts\\Database\\ModelIdentifier\":4:{s:5:\"class\";s:12:\"App\\Messages\";s:2:\"id\";i:36;s:9:\"relations\";a:0:{}s:10:\"connection\";s:5:\"mysql\";}s:7:\"counter\";i:1;s:3:\"job\";N;s:10:\"connection\";N;s:5:\"queue\";N;s:15:\"chainConnection\";N;s:10:\"chainQueue\";N;s:5:\"delay\";N;s:10:\"middleware\";a:0:{}s:7:\"chained\";a:0:{}}","commandName":"App\\Jobs\\ESender"},"displayName":"App\\Jobs\\ESender","timeoutAt":1594841911,"maxExceptions":null,"maxTries":null,"job":"Illuminate\\Queue\\CallQueuedHandler@call","delay":null,"attempts":1}

see the cmd error here.

parts of code:

#1

class ESender implements ShouldQueue
{
    use Dispatchable, InteractsWithQueue, Queueable, SerializesModels;


    /**
     * The number of times the job may be attempted.
     *
     * @var int
     */
    public $tries = 100;

    /**
     * The number of seconds the job can run before timing out.
     *
     * @var int
     */
    public $timeout = 9999999;

     ...more code...
}

#2

public function handle(){
    Redis::throttle('key')->allow(1)->every(20)->then(function () {
         //send email
           ..... more code .....

        }, function () {
            // Could not obtain lock...
            return $this->release(10);
        });
    }

and this is my configuration:

queue.php:

'redis' => [
            'driver' => 'redis',
            'connection' => 'default',
            'queue' => env('REDIS_QUEUE', 'default'),
            'retry_after' => 9000,
            'block_for' => null,
        ],

.env

BROADCAST_DRIVER=log
CACHE_DRIVER=file
QUEUE_CONNECTION=database
SESSION_DRIVER=file
SESSION_LIFETIME=300
REDIS_CLIENT = predis
REDIS_HOST=127.0.0.1
REDIS_PASSWORD=null
REDIS_PORT=6379
QUEUE_DRIVER=database
Fadi Sharif
  • 300
  • 1
  • 3
  • 11
  • could you add a command that Queue was trying to call please. Something is failing in your scripts and we would need to know more to help you out. – Vlad Vladimir Hercules Jul 15 '20 at 19:00
  • @VladVladimirHercules its work fine with number of queues (who are at first). But then these errors appear. You can review the image above. – Fadi Sharif Jul 15 '20 at 19:13
  • I was update is how command was called in your failed_jobs table. What you sent us is an error, I am after the actually command call. Also, did that error include any other information? – Vlad Vladimir Hercules Jul 15 '20 at 19:43
  • The timeout and tries is set for an individual job, so when your job is queued that job has, based on your code, 100 tries to execute. – Vlad Vladimir Hercules Jul 15 '20 at 19:44
  • @VladVladimirHercules The question has been updated under the payload section,please check – Fadi Sharif Jul 15 '20 at 19:54
  • Could you delete all code in the handlers, after that do you still get failed attempts? – Vlad Vladimir Hercules Jul 15 '20 at 19:58
  • And there are no other errors in the logs? Is it failing every time after the first 10? – Maarten Veerman Jul 15 '20 at 19:59
  • - I don't think the problem is inside the handlers because it works with the first queues. - It is true that failure occurs after 10. @VladVladimirHercules – Fadi Sharif Jul 15 '20 at 20:13
  • @FadiSharif did you try running it with an empty handler? – Vlad Vladimir Hercules Jul 15 '20 at 20:16
  • @VladVladimirHercules yes, and i have same problem. – Fadi Sharif Jul 15 '20 at 20:25
  • @FadiSharif do any of your queue jobs work? – Vlad Vladimir Hercules Jul 15 '20 at 20:26
  • @FadiSharif does queue work using any other queue drivers, i.e. database? Right now it looks as if you have misconfigured the queue or made custom modifications which we are not aware of. Also in your .env you are using QUEUE_DRIVER=database – Vlad Vladimir Hercules Jul 15 '20 at 20:30
  • The early queue is working properly but then errors appear. I changed QUEUE_CONNECTION and QUEUE_DRIVER for the purpose of testing, but I am already working on redis. On the whole, the operation did not succeed on both (redis and database) – Fadi Sharif Jul 15 '20 at 20:39
  • @FadiSharif its worth trying to run the queue on a blank laravel install to see what will happen. I would also, personally, start debugging laravel's code, starting with `laravel\framework\src\Illuminate\Queue\Worker.php:649`; adding some basic Log::error('something here') should give you an idea of what is going on. xDebug may come handy in this type of situations. – Vlad Vladimir Hercules Jul 16 '20 at 07:04
  • Having the same issue. Any solution? – Utku Dalmaz Mar 07 '22 at 13:10

4 Answers4

7

You set a timeout in your job, but this timeout is larger than the value in retry_after which you have defined in the this config.

See https://laravel.com/docs/7.x/queues#job-expirations-and-timeouts

There is a clear warning:

The --timeout value should always be at least several seconds shorter than your retry_after configuration value. This will ensure that a worker processing a given job is always killed before the job is retried. If your --timeout option is longer than your retry_after configuration value, your jobs may be processed twice.

You could define a new connection for long running jobs, and set this connection on the job (dispatch to specific connection), instead of using the timeout.

Maarten Veerman
  • 1,546
  • 1
  • 8
  • 10
5

Just increase the timeout if you want ,But be careful not to occupy server resources for long periods

php artisan queue:work --timeout=10000000
Fadi Sharif
  • 300
  • 1
  • 3
  • 11
  • My question, do we need to keep restarting the above queue work after the timeout? I mean can it detect the queue is empty then don't run. but as soon as the queue is filled with any record it should auto restart? – Amit Shah Apr 28 '23 at 08:32
2

The command, that runs your queue worker needs --tries= and --timeout= to set the out limits your queue worker to allow.

This makes sure that your commands cannot go beyond the limits of your defined workers.

You can use the job properties to achieve timeout or tries, below. And use queue configuration file to set a default.

Danny Ebbers
  • 919
  • 5
  • 11
  • Unfortunately I did this but it didn't work, you can notice that I did it inside the class ESender – Fadi Sharif Jul 15 '20 at 19:44
  • Probably not your solution but your .env file says connection database, but you posted queue.php focuses on redis?.. What are you actually using.. did you try to switch? You have the same issue with both drivers? – Danny Ebbers Jul 15 '20 at 19:49
  • Are you aware that your throttle / release will increase the number of attempts as well... see https://github.com/laravel/ideas/issues/1381 Doesn't make much sense, but it does increase the attempts.. – Danny Ebbers Jul 15 '20 at 19:52
  • I changed QUEUE_CONNECTION and QUEUE_DRIVER for the purpose of testing, but I am already working on redis. On the whole, the operation did not succeed on both – Fadi Sharif Jul 15 '20 at 20:02
1

Run

php artisan config:clear
php artisan optimization:clear

restart supervisor

COgroup Digital
  • 19
  • 1