PHP using fwrite and fread with input stream

Question

I'm looking for the most efficient way to write the contents of the PHP input stream to disk, without using much of the memory that is granted to the PHP script. For example, if the max file size that can be uploaded is 1 GB but PHP only has 32 MB of memory.

define('MAX_FILE_LEN', 1073741824); // 1 GB in bytes
$hSource = fopen('php://input', 'r');
$hDest = fopen(UPLOADS_DIR.'/'.$MyTempName.'.tmp', 'w');
fwrite($hDest, fread($hSource, MAX_FILE_LEN));
fclose($hDest);
fclose($hSource);

Does fread inside an fwrite like the above code shows mean that the entire file will be loaded into memory?

For doing the opposite (writing a file to the output stream), PHP offers a function called fpassthru which I believe does not hold the contents of the file in the PHP script's memory.

I'm looking for something similar but in reverse (writing from input stream to file). Thank you for any assistance you can give.

score 9 · Accepted Answer · answered Aug 28 '13 at 20:24

9

Yep - fread used in that way would read up to 1 GB into a string first, and then write that back out via fwrite. PHP just isn't smart enough to create a memory-efficient pipe for you.

I would try something akin to the following:

$hSource = fopen('php://input', 'r');
$hDest = fopen(UPLOADS_DIR . '/' . $MyTempName . '.tmp', 'w');
while (!feof($hSource)) {
    /*  
     *  I'm going to read in 1K chunks. You could make this 
     *  larger, but as a rule of thumb I'd keep it to 1/4 of 
     *  your php memory_limit.
     */
    $chunk = fread($hSource, 1024);
    fwrite($hDest, $chunk);
}
fclose($hSource);
fclose($hDest);

If you wanted to be really picky, you could also unset($chunk); within the loop after fwrite to absolutely ensure that PHP frees up the memory - but that shouldn't be necessary, as the next loop will overwrite whatever memory is being used by $chunk at that time.

answered Aug 28 '13 at 20:24

pauld

231
2
3

Thanks pauld, I like your idea of using unset too. – Lakey Aug 30 '13 at 02:39
2

@pauld Nice! For best performance, the unset should be used more like `while (...) {...} unset($chunk);` re-allocating the memory every iteration wouldn't have much of an impact, but freeing that memory _after_, is a bit more important here. – Tony Chiboucas Sep 24 '14 at 18:56
@TonyChiboucas Why would you free 1024 bytes of memory? – marijnz0r Apr 05 '17 at 15:02
@marijnz0r because this can be happening in the middle of a much longer PHP thread, among thousands of parallel requests. Leaving aside the potential memory leaks, and the opening to slow-loris, in a production environment (as part of a logger for instance) this could quickly become MB of memory. The most important thing to remember about performance and memory management, is that every byte matters. They add up. – Tony Chiboucas Jun 01 '17 at 21:00
@TonyChiboucas fair answer, but don't forget about premature optimization! – marijnz0r Jun 02 '17 at 13:26
1

@marijnz0r, following a good practice isn't premature optimization. Any code which operates on memory like this should ALWAYS free the memory. Premature optimization would be to analyze the performance of this code, and some variations, selecting an implementation with the lowest time to completion on large data-sets. "Free the memory" isn't early optimization. – Tony Chiboucas Aug 24 '17 at 20:54
Set `$chunk` to `null` before unsetting, otherwise you're still storing it's contents in GC – zanderwar May 14 '22 at 02:21
1

@TonyChiboucas picture an API that receives 30 million requests per hour, then question why you would free 1024 from memory :) – zanderwar May 14 '22 at 02:22
1

@zanderwar, that's exactly the point I was making ^_^ – Tony Chiboucas May 25 '22 at 23:13

PHP using fwrite and fread with input stream

1 Answers1

Linked