0

For a customer, I need to upload a cvs file. The file has nearly 35000 lines. I used maatwebsite/excel package.

Excel::filter('chunk')->load($file->getRealPath())->chunk(100, 
    function($results) {
        foreach ($results as $row) {
            // Doing the  import in the DB
        }
    }
});

I can't change the max_execution_time because our server doesn't allow executions more than 300 seconds.

I tried also tried another way without any package but that failed also.

$csv = utf8_encode(file_get_contents($file));
$array = explode("\n", $csv);

foreach ($array as $key => $row) {
    if($key == 0) {
        $head = explode(',', $row);

        foreach ($head as $k => $item) {
            $h[$key][] = str_replace(' ', '_', $item);
        }
    }
    if($key != 0) {
        $product = explode(',', $row);

        foreach ($product as $k => $item) {
            if($k < 21)
                $temp[$key][$h[0][$k]] = $item;
        }
    }
}

foreach ($temp as $key => $value) {
    // Doing the  import in the DB
}

Does anyone have an idea?

Edit:

So I made an artisan command. When I execute this in terminal it get's executed and all 35000 rows are imported. Thanks to common sence.

I just can't figure out how to make the command run asynchrone so the user can close his browser. Can anyone explain how to get that done?

Andreea Ilie
  • 65
  • 1
  • 1
  • 7
  • Why don't you try to do a mass upload at the end? imagine, instead of saving it to DB on the foreach, build a large array and in the end insert all at once. https://stackoverflow.com/questions/12702812/bulk-insertion-in-laravel-using-eloquent-orm – Sérgio Reis Jan 17 '18 at 10:30
  • Another approach is to throw `import to database process` chunk to queue process. – Dharma Saputra Jan 17 '18 at 10:32
  • Have a look at my answer there: https://stackoverflow.com/questions/48174355/laravel-insert-millions-of-database-rows-from-models/48176164#48176164 – common sense Jan 17 '18 at 12:06
  • _“how to make the command run asynchrone so the user can close his browser”_ - you could try `ignore_user_abort`, http://php.net/manual/en/features.connection-handling.php (but this will still be affected by the max execution time.) – CBroe Jan 18 '18 at 15:18
  • I need a way so the artisan command can be executed without intervention of the user. The only thing the user must do is upload a file. The rest should be automated and I can't get a timeout. – Andreea Ilie Jan 18 '18 at 15:33

1 Answers1

0

Remember that it will take some time for any file (particularly if it is large) to be uploaded to the server via the user's web browser, so you definitely do not want to inadvertently encourage your users to close their web browser before the file has been completely uploaded.

Possibly you may be able to update your code so that it displays a confirmation message to the user after the file has been uploaded, but before it has been processed.

However, I do not know whether closing the browser at that point would actually terminate the script immediately (or whether it would continue to completion), or whether instead you would need to invoke a separate program on the server (perhaps a cron job running every few minutes) to parse any newly uploaded files, as a separate task?

(Incidentally, please be aware that because of the way that the StackExchange Q&A format works, it is strongly preferred that you should have made your further response "answer" in this question as an edit to your original post, rather than an "answer" (which it is not).

StackExchange is not like an older "linear" forum: amendments or updates to the original question should be made to the original question itself, and answer posts should be used literally only for actual suggested answers to the question. (And this last aside from myself should really have been a "comment", but unfortunately I do not yet have enough reputation points to do so.))

dave559
  • 111
  • 1
  • 9
  • I know it takes some time to upload the file but that is not the issue. The file is uploaded and saved as it should be. File::move($file->getRealPath(), 'storage/upload.csv'); Artisan::call('csv:import'); The artisan command is executed but that takes to long so I get a timeout. I need to execute that command async so I wont get a timeout. When I run te command in terminal it get's executed and I don't get a timeout. – Andreea Ilie Jan 18 '18 at 15:04