Lumen Database Queue first job always failing Allowed memory exhausted

Question

I have a very odd situation where I set up a job to run in my Lumen database queue and all but the first job is processed. I do keep getting this particular error:

[2017-12-12 22:07:10] lumen.ERROR: Symfony\Component\Debug\Exception\FatalErrorException: Allowed memory size of 1073741824 bytes exhausted (tried to allocate 702558208 bytes) in /var/www/vhosts/XXXXXXXXX$
Stack trace:
#0 /var/www/vhosts/XXXXXXXX/vendor/laravel/lumen-framework/src/Concerns/RegistersExceptionHandlers.php(54): Laravel\Lumen\Application->handleShutdown()
#1 [internal function]: Laravel\Lumen\Application->Laravel\Lumen\Concerns\{closure}()
#2 {main}

I have tried allowing the memory limit to go up but I keep getting the same error with differing values for the exhausted memory.

I find it very odd that it is always the first job and all of the rest of the jobs run perfectly fine. Should I be looking for bad data in the first job?

My code basically looks like this:

This is my Command file

namespace App\Console\Commands;

use App\Jobs\UpdateNNNAppListJob;
use Illuminate\Console\Command;
use App\Services\MiddlewareApi;
use Illuminate\Support\Facades\DB;
use Illuminate\Support\Facades\Log;
use Mockery\Exception;
use Illuminate\Support\Facades\Queue;

class AddEmailsToAppList extends Command
{
    /**
     * The name and signature of the console command.
     *
     * @var string
     */
    protected $signature = 'addemails:nnnmobileapp';

    /**
     * The console command description.
     *
     * @var string
     */
    protected $description = 'This will add all mobile app users in the database to the nnn mobile app list.';

    /**
     * Create a new command instance.
     *
     * @return void
     */
    public function __construct()
    {
        parent::__construct();
    }

    public function handle()
    {
        $chunkSize = 500; //this is the most middleware can handle with its bulk signup call
        $emailChunks = $this->getEmailsToAdd($chunkSize);
        $jobDelay = 120; //time between queued jobs
        $jobDelayTimeKeeper = 60; //This will be the actual time delay that will be put into the later method

        foreach ($emailChunks as $emailChunk) {
            Queue::later($jobDelayTimeKeeper, new UpdateMmpAppListJob($emailChunk));
            $jobDelayTimeKeeper = $jobDelayTimeKeeper + $jobDelay;
        }
    }

    public function getEmailsToAdd($chunkSize)
    {
        $emails = DB::table('app_users')
            ->join('app_datas', 'app_datas.customer_number', '=', 'app_users.customer_number')
            ->select('app_users.email')
            ->get()
            ->chunk($chunkSize);

        return $emails;
    }
}

Here is my Job File

<?php
namespace App\Jobs;

use App\Services\MiddlewareApi;
use Illuminate\Support\Facades\Log;
use Mockery\Exception;

class UpdateMmpAppListJob extends Job
{

    /**
     * Array of emails to update list with
     * @var array
     */
    protected $emailArray;

    /**
     * The number of times the job may be attempted.
     *
     * @var int
     */
    public $tries = 2;

    public function __construct($emailArray)
    {
        $this->emailArray = $emailArray;
    }

    public function handle()
    {
        $listCodeToAddTo = 'NNNAPP';
        $sourceId = 'NNNNNNN';

        $middlewareApi = new MiddlewareApi();

        try {
            $middlewareApi->post_add_customer_signup_bulk($listCodeToAddTo, $this->emailArray, $sourceId);
        } catch (\Exception $e) {
            Log::error('An error occurred with theUpdateMmpAppListJob: ' . $e);
            mail('djarrin@NNN.com', 'UpdateNnnAppListJob Failure', 'A failure in the UpdateNnnAppListJob, here is the exception: ' . $e);
        }

    }

    public function failed(\Exception $exception)
    {
        mail('djarrin@moneymappress.com', 'Push Processor Que Failure', 'A failure in the UpdateMmpAppListJob, here is the exception: ' . $exception);
    }
}

Any help/suggestions on this issue would be appreciate.

Answer 1

Your code calls ->get() which will load the entire result into memory. This causes the huge memory allocation that you're seeing. Remove it and let ->chunk(...) work with the database builder instead of the in-memory Collection that get() has returned. You would also have to provide a callback to chunk that processes every chunk.

public function handle() {
    $chunkSize = 500; //this is the most middleware can handle with its bulk signup call
    $jobDelay = 120; //time between queued jobs
    $jobDelayTimeKeeper = 60; //This will be the actual time delay that will be put into the later method

    DB::table('app_users')
        ->join('app_datas', 'app_datas.customer_number', '=', 'app_users.customer_number')
        ->select('app_users.email')
        ->chunk($chunkSize, function($emailChunk) use (&$jobDelayTimeKeeper, $jobDelay) {
            Queue::later($jobDelayTimeKeeper, new UpdateMmpAppListJob($emailChunk));
            $jobDelayTimeKeeper = $jobDelayTimeKeeper + $jobDelay;
        });
}

Answer 2

The above concept is correct but this syntax was required to get past the

[2017-12-14 22:08:26] lumen.ERROR: RuntimeException: You must specify an orderBy clause when using this function. in /home/vagrant/sites/nnn/vendor/illuminate/database/Query/Builder.php:1877

This is for Lumen 5.5:

public function handle()
    {
        $chunkSize = 500; //this is the most middleware can handle with its bulk signup call
        $jobDelay = 120; //time between queued jobs
        $jobDelayTimeKeeper = 60; //This will be the actual time delay that will be put into the later method

        $emails = DB::table('app_users')
            ->join('app_datas', 'app_datas.customer_number', '=', 'app_users.customer_number')
            ->select('app_users.email')->orderBy('app_users.id', 'desc');

        $emails->chunk($chunkSize, function($emailChunk) use (&$jobDelayTimeKeeper, $jobDelay) {
            Queue::later($jobDelayTimeKeeper, new UpdateMmpAppListJob($emailChunk));
            $jobDelayTimeKeeper = $jobDelayTimeKeeper + $jobDelay;
        });
    }