0

I am inserting a big number of lines into my db. I have to split my inserts in batches because of maximal json parameter length.

Please consider this original snippet

/** @var $this->intermediateSchedule ScheduleItem[] */
/** @var $this->insertStatement Doctrine\DBAL\Statement */
$FBS = 50000;
for($i = 0 ; $i < count($this->intermediateSchedule); $i += $FBS) {
            $batch = array_values(array_slice($this->intermediateSchedule, $i, $FBS));
            $jsonstr = json_encode($batch);
            $start = Carbon::now();
            $this->insertStatement->executeStatement(['jsonstr' => $jsonstr]);
            $seconds= $start->floatDiffInSeconds();
            $flushtime = number_format(1000.0 * $seconds,2);
            $usperrow = number_format((1000000.0 * $seconds) / array_reduce($batch, fn ($carry, $item) => $carry + count($item->periods), 0), 2);
            $jsonsize = Helper::formatMemory(strlen($jsonstr));
            $this->output("Flush took $flushtime ms for $jsonsize, $usperrow us/row");
        }

and

class ScheduleItem implements \JsonSerializable
{
    public $dc;
    public $level;
    public $periods;
}

I am running through this loop several times successfully but it seems to leak somewhere, see this output:

108871 Initial stack size 12141, UoW size: 980243, mem 5,9 GiB
108871 
108871 Flush took 6,526.84 ms for 63,8 MiB, 1.74 us/row
108871 
108871 Flush took 6,458.60 ms for 63,8 MiB, 1.72 us/row
108871 
108871 Flush took 6,612.99 ms for 63,2 MiB, 1.78 us/row
108871 
108871 Flush took 6,377.19 ms for 63,0 MiB, 1.72 us/row
108871 
108871 Flush took 6,512.89 ms for 63,3 MiB, 1.75 us/row
108871 
108871 Flush took 6,892.72 ms for 63,3 MiB, 1.85 us/row
108871 
108871 Flush took 6,817.15 ms for 63,3 MiB, 1.83 us/row
108871 
108871 Flush took 6,946.34 ms for 63,3 MiB, 1.87 us/row
108871 
108871 Flush took 7,006.81 ms for 63,3 MiB, 1.88 us/row
108871 
108871 Flush took 6,912.38 ms for 63,3 MiB, 1.86 us/row
108871 
108871 
108871 Fatal error: Allowed memory size of 15032385536 bytes exhausted (tried to allocate 62918656 bytes) in /var/www/xxx/src/Service/DataCa
tegorySchedulerHelper.php on line 177
108871

I use PHP 7.4 and Doctrine DBAL 2.13.1.

olidem
  • 1,961
  • 2
  • 20
  • 45
  • See https://stackoverflow.com/questions/26616861/memory-leak-when-executing-doctrine-query-in-loop/43916144 – emptyhua Oct 03 '21 at 04:43
  • Thanks, but this code is already running in production environment, so `--no-debug` is set, also there is no logger attached, `$this->em->getConnection()->getConfiguration()->getSQLLogger()` returns null – olidem Oct 03 '21 at 15:35

0 Answers0