Laravel Out of memory on localhost when trying to upload very big file

Question

I am trying to upload csv file so that I can upload those data into the database. The code for the same is as bellow

public function upload(Request $request){
        set_time_limit(0);
        ini_set('MAX_EXECUTION_TIME', 36000);
        $file = Input::file('file');
        $filePath = $file->getRealPath();

       $handle = fopen($filePath, "r");
        while(!feof($handle))
        {
           << DO THE DATABASE OPERATION >>
        }
        fclose($handle);

  return redirect()->back()->with('success', 'File uploaded successfully');
}

This works fine if the file size is less lets say about 100 or 200mb. But when the file size is big like 2GB. It closes the local server. In the console, it says out of memory

My php.ini settings are :

post_max_size=10000M
upload_max_filesize=10000M

My system config:

Window machine 64 bit

Problem

It's not only failing to upload but also closing the development server i.e localhost:8000

Can anyone please tell me why this is happening and how can i fix it. I did follow a couple of threads on StackOverflow like this

phpMyAdmin: Can't import huge database file, any suggestions?

Large file uploads failing php

Laravel out of memory issue?

Uploading a file larger than 2GB using PHP

But unfortunately, these solutions did not help.

Possible duplicate of [Uploading a file larger than 2GB using PHP](https://stackoverflow.com/questions/4614147/uploading-a-file-larger-than-2gb-using-php) — Chris Forrence, Aug 28 '18 at 14:45
Yes same question but those solutions didn't work. Please check my php.ini settings. I did apply those solutions already. and the file I m trying to upload is 1.6GB so that doesn't make any sense why the application turning off the local server. — user7747472, Aug 28 '18 at 14:48
That was pretty quick to edit your Apache configuration ;) What version of PHP are you running? And is the file 2GB or 1.6GB? — Chris Forrence, Aug 28 '18 at 14:51
That was not quick edit, just in case u have not noticed, I added the php.ini setting at the time of posting this question. I am using `php 7.1` . The file i am trying to upload is 1.6GB and i have another file which is 4.3GB. I want to upload both of them — user7747472, Aug 28 '18 at 14:55
Since you're already essentially allowing it to have unlimited memory, why not set it to -1? — Chris Forrence, Aug 28 '18 at 15:07
I took your suggestions set `memory_limit=-1` , restarted the apache server , and run the application but the problem still persists. I tried to upload big file with same config before but that was in core php that worked fine . So i m guessing there might be some settings in laravel application itself. — user7747472, Aug 28 '18 at 15:15

score 0 · Answer 1 · answered Aug 28 '18 at 15:12

I would check your Windows Event Viewer to try and track down why the server is crashing. That would hopefully give you some insight into what the issue is.

If you can't find the source of the crash and since you are using 7.1 and large files, try processing the file with a generator. You're loading the whole thing into memory. Even with that large value for memory_limit, I still think that loading that whole file into memory might be your issue.

http://php.net/manual/en/language.generators.php

This will stream it line by line, but you can find other examples that allow you to chunk the file.

public function read_file($file)
{
    $fp = fopen($file, 'r');

    while(($line = fgets($fp)) !== false)
        yield $line;

    fclose($fp);
}  

public function upload(Request $request){
            set_time_limit(0);
            ini_set('MAX_EXECUTION_TIME', 36000);
            $file = Input::file('file');
            $filePath = $file->getRealPath();

            foreach(read_file($filePath) as $line)
            {
               << DO THE DATABASE OPERATION >>
            }

      return redirect()->back()->with('success', 'File uploaded successfully');
    }

Thank you erik for the suggestions .I will try that out :)) – user7747472 Aug 28 '18 at 15:16 — user7747472, Aug 28 '18 at 15:16

Laravel Out of memory on localhost when trying to upload very big file

1 Answers1