Laravel Solution for "Allowed memory size of 134217728 bytes exhausted" Error in Storage::append()

Question

I implemented a code with Laravel to handle chunk uploading, which can be seen below.

// if total size equals length of file we have gathered all patch files
if ($size == $length) {
    // write patches to file
    foreach ($patch as $filename) {
        // get offset from filename
        list($dir, $offset) = explode('.patch.', $filename, 2);

        // read patch and close
        $patch_contents = Storage::disk('tmp')->get($filename);

        // apply patch
        Storage::disk('tmp')->append($dir . $name, $patch_contents, "");
    }

    // remove patches
    foreach ($patch as $filename) {
        Storage::disk('tmp')->delete($filename);
    }
}

The problem is that the following error occurs for large files.

"Allowed memory size of 134217728 bytes exhausted (tried to allocate 160000008 bytes)"

I know that the error is related to the append method. I solved the problem according to the solution in this link and as follows.

// if total size equals length of file we have gathered all patch files
if ($size == $length) {
    $time_limit = ini_get('max_execution_time');
    $memory_limit = ini_get('memory_limit');

    set_time_limit(0);
    ini_set('memory_limit', '-1');

    // write patches to file
    foreach ($patch as $filename) {

        // get offset from filename
        list($dir, $offset) = explode('.patch.', $filename, 2);

        // read patch and close
        $patch_contents = Storage::disk('tmp')->get($filename);

        // apply patch
        Storage::disk('tmp')->append($dir . $name, $patch_contents, "");
    }

    // remove patches
    foreach ($patch as $filename) {
        Storage::disk('tmp')->delete($filename);
    }

    set_time_limit($time_limit);
    ini_set('memory_limit', $memory_limit);
}

But I don't have a good feeling about this solution! My question is,

first of all, why does the append method cause such an error?
Is the solution appropriate?
On the other hand, what is Laravel's solution to this problem?

apokryfos · Accepted Answer · 2020-05-16T06:38:12.103

The root cause seeems to be in the source of append.

$this->put($path, $this->get($path).$separator.$data);

This gets the file contents and then concatenates the data and puts the file back. Not sure why this is done this way but I'm guessing it's because opening files in append mode is not supported in all storage types and Storage is obliged to implement CloudFilesystemContract meaning that it's meant to work for cloud storage (where you generally can't "append" to a file).

Digging a bit deeper one can find that Laravel's Storage "driver" is backed by Flysystem and that does not include an append functionality in its interface so Laravel is obliged to implement one only using the interface methods provided.

You can always just roll your own solution:

// if total size equals length of file we have gathered all patch files
if ($size == $length) {
    // write patches to file
    foreach ($patch as $filename) {
        // get offset from filename
        list($dir, $offset) = explode('.patch.', $filename, 2);

        // read patch and close
        $patch_contents = Storage::disk('tmp')->get($filename);

        // apply patch
        $fhandle = fopen($dir.$name, "a"); // You may need to adjust the filename 
        fwrite($fhandle, $patch_contents);
        fclose($fhandle);
    }

    // remove patches
    foreach ($patch as $filename) {
        Storage::disk('tmp')->delete($filename);
    }
}

Laravel Solution for "Allowed memory size of 134217728 bytes exhausted" Error in Storage::append()

1 Answers1