0

I'm trying to import 10,000 records in excel with the library "Maatwebsite / Laravel-Excel" in Laravel, but when importing 10,000 records, the page returns a 504 timeout error (the odd thing is that it keeps inserting data in the db after that). Well then, I show you my code to see if you can help me, thanks.

This is the Import Inventory, where I programmed everything I should do with the data to be imported into the db.

class InventarioImport implements ToCollection, WithChunkReading
{
    use Importable;
    /**
     * @param array $row
     *
     * @return User|null
     */

    public function chunkSize(): int
    {
        return 250;
    }


    public function collection(collection $rows)
    {
         foreach ($rows as $row)
        {

        $almacenes = Almacen::where('tienda_id', \Session::get('tienda_id'))->get();
        $codigos = Codigo::where('tienda_id', \Session::get('tienda_id'))->get();
        $contc=-1;
        $conta=0;

        foreach ($codigos as $codigo) {
            $contc++;
        }
        foreach ($almacenes as $almacen) {
            $conta++;
        }
        if($row[$contc+1]=="DESCRIPCION") {
                continue 1;

            }

        $idmarca = Marca::where('nombre_marca', $row[$contc+2])->where('tienda_id', \Session::get('tienda_id'))->first();

        if ($idmarca==null) {
           $marcaid= Marca::create([
           'nombre_marca'     => $row[$contc+2],
           'tienda_id'     => \Session::get('tienda_id'),
           'estado'     => "A",

        ])->id;
}
else {
  $marcaid = $idmarca->id;
}
$buscarcategoria = Categoria::where('nombre_categoria', $row[$contc+3])->where('tienda_id', \Session::get('tienda_id'))->first();
        if ($buscarcategoria==null) {
           Categoria::create([
           'nombre_categoria'     => $row[$contc+3],
           'tienda_id'     => \Session::get('tienda_id'),
           'estado'     => "A",

        ]);
}
$i=0;
       $buscaritem = Item::where('nombre_item', $row[$contc+1])->where('tienda_id', \Session::get('tienda_id'))->first();

 $buscarunidad = Unidad::where('nombre_unidad', $row[$contc+4])->first();


   $buscarmoneda = ConfiguracionMoneda::where('abreviacion_moneda', $row[$conta+3+$contc+4])->where('tienda_id', \Session::get('tienda_id'))->first();
    $itemid= Item::create([
            'marca_id' => $marcaid,
           'nombre_item'     => $row[$contc+1],
           'unidad_id'     => $buscarunidad->id,
           'stock_minimo'     => $row[$conta+1+$contc+4],
           'stock_maximo'     => $row[$conta+2+$contc+4],
           'moneda_id'     => $buscarmoneda->id,
           'precio'     => $row[$conta+4+$contc+4],
           'impuesto'     => $row[$conta+5+$contc+4],
           'margen_final'     => $row[$conta+6+$contc+4],
           'margen_inicio'     => $row[$conta+7+$contc+4],
           'notas'     => $row[$conta+8+$contc+4],
           'estado'     => "A",
           'tienda_id'     => \Session::get('tienda_id'),

        ])->id;
        $a=$contc+4;
        $j=$a+1;
        foreach ($almacenes as $almacen) {

       $buscaralmacen = Almacen::where('nombre_almacen', $almacen->nombre_almacen)->where('tienda_id', \Session::get('tienda_id'))->first();

          ItemStock::create([
           'item_id'     => $itemid,
           'almacen_id' => $buscaralmacen->id,
           'detalle' => "Saldo Inicial",
            'cantidad' => $row[$j],
            'tipo' => "M",

           'tienda_id'     => \Session::get('tienda_id'),
           'estado'     => "A",
        ]);


        $kardex = new Kardex();
                $kardex->item_id = $itemid;
                $kardex->fecha = date("Y/m/d");
                $kardex->operacion = "Inicial";
 $kardex->tipo = "";
 $kardex->serie = "";
 $kardex->numero = "";

                $kardex->almacen_id = $buscaralmacen->id;
                $kardex->tienda_id = \Session::get('tienda_id');
                $kardex->saldocantidad = $row[$j];
                $kardex->saldocosto = $row[$conta+4+$contc+4];
                $kardex->saldototal = $row[$conta+4+$contc+4]*$row[$j];

                $kardex->save();
                $j++;
 }

  foreach ($codigos as $codigo) {

       $buscarcodigo = Codigo::where('nombre_codigo', $codigo->nombre_codigo)->where('tienda_id', \Session::get('tienda_id'))->first();
       $buscaritem = Item::where('nombre_item', $row[$contc+1])->where('tienda_id', \Session::get('tienda_id'))->first();

          ItemCodigo::create([
           'item_codigo'     => $row[$i],
           'codigo_id' => $buscarcodigo->id,
           'item_id' => $buscaritem->id,
           'tienda_id'     => \Session::get('tienda_id'),
           'estado'     => "A",
        ]);
        $i++;


 }


    }
    }
}

This is my function in Controller, which I do the import as indicated in the previous script.

public function subirinventario(Request $request)
    {
        $title = 'Inventario subido';

        $contarr= $request->contarr;
  $arrayitems = Excel::import(new InventarioImport, $request->path);
        return view('item.finalizar', compact('contarr','title'));
    }

That would be all, I missed the routes and the views, but it works fine with few records, I have to emphasize that I have increased the runtime of nginx, restarted and nothing.

This is an image with the preview of the data and columns, only 10 are shown, but more than 10 thousand were detected.

  • Long-running scripts such as large inserts is better done via a command line or offloaded to a queue. The server or web browser will return the timeout even though the script will still run. – aynber Feb 18 '20 at 18:47
  • Hello, thanks for your response, I am a little novice with this of the servers, if I have configured on my server the runtime of say 20 minutes and for 10,000 records it does not require even 5 minutes, why does the timeout come out? I do not understand that. – Vicente Fuentes Feb 18 '20 at 18:58
  • Can you post your nginx and vhost configuration? and anything related to fastcgi_* and proxy_* directives. – Raul Feb 19 '20 at 01:18
  • Also, that code needs to be refactored - the number of queries and inefficiency there will cause it to take a long time to process. You can try to split the doc maybe, queue it up for processing, read in batches, etc. But no matter what settings you end up using in nginx, at one point they won't help, as this process will take 10 minutes to finish for 50k rows. So you need to optimize the code. Start with queue, then batches. And then find a way to display a progress in the interface. – Raul Feb 19 '20 at 01:28

1 Answers1

0

You can try to put this 2 lines in the start of your method/function:

    ini_set('memory_limit','1024M');
    set_time_limit(3000000); //You can use 0 to remove limits

As I stated above you can use 0 inside the set_time_limit method, but i don't recommend this, because losing all the control of your execution is not worth in most cases

Hope it works!

ItsEdgar94
  • 344
  • 1
  • 8
  • Ok, should definitely because all the foreach you are doing in the collection method of your import class, have you tried to put those lines inside it? – ItsEdgar94 Feb 18 '20 at 20:00
  • yes I put it inside the Importer and also in the Controller, but I'm still getting error 504. – Vicente Fuentes Feb 18 '20 at 20:17
  • Ups, sorry, didn't read you are using nginx... ^.^' https://stackoverflow.com/questions/16002268/prevent-nginx-504-gateway-timeout-using-php-set-time-limit maybe this post best answer can help? – ItsEdgar94 Feb 18 '20 at 20:49
  • Thank you for continuing with me on this, I still have not been able to make it work, I have configured nginx with everything mentioned in the post, restarted and nothing. = ( – Vicente Fuentes Feb 18 '20 at 21:18