16

I have a PHP script that occasionally needs to write large files to disk. Using file_put_contents(), if the file is large enough (in this case around 2 MB), the PHP script runs out of memory (PHP Fatal error: Allowed memory size of ######## bytes exhausted). I know I could just increase the memory limit, but that doesn't seem like a full solution to me--there has to be a better way, right?

What is the best way to write a large file to disk in PHP?

hakre
  • 193,403
  • 52
  • 435
  • 836
Joe Lencioni
  • 10,231
  • 18
  • 55
  • 66
  • 1
    What's the source of the data? Also, most PHP configs should be able to handle 2MB's with no problem. Your memory limit may be way under. – webbiedave Jan 25 '11 at 19:40
  • 1
    Where does the file come from? If it exists on disk already, then `copy()` would be most appropriate. – mario Jan 25 '11 at 19:40
  • In this case, I am getting the file from a remote server using curl. So, I have it in a variable. – Joe Lencioni Jan 25 '11 at 19:43
  • Skip curl, use `copy("http://example.com/file", "/tmp/local.txt")` – mario Jan 25 '11 at 20:12
  • @mario: that would be pretty easy, but I am copying multiple files and I like that I can run them simultaneously using curl_multi_* stuff – Joe Lencioni Feb 01 '11 at 19:57

3 Answers3

19

You'll need a temporary file in which you put bits of the source file plus what's to be appended:

$sp = fopen('source', 'r');
$op = fopen('tempfile', 'w');

while (!feof($sp)) {
   $buffer = fread($sp, 512);  // use a buffer of 512 bytes
   fwrite($op, $buffer);
}

// append new data
fwrite($op, $new_data);    

// close handles
fclose($op);
fclose($sp);

// make temporary file the new source
rename('tempfile', 'source');

That way, the whole contents of source aren't read into memory. When using cURL, you might omit setting CURLOPT_RETURNTRANSFER and instead, add an output buffer that writes to a temporary file:

function write_temp($buffer) {
     global $handle;
     fwrite($handle, $buffer);
     return '';   // return EMPTY string, so nothing's internally buffered
}

$handle = fopen('tempfile', 'w');
ob_start('write_temp');

$curl_handle = curl_init('http://example.com/');
curl_setopt($curl_handle, CURLOPT_BUFFERSIZE, 512);
curl_exec($curl_handle);

ob_end_clean();
fclose($handle);

It seems as though I always miss the obvious. As pointed out by Marc, there's CURLOPT_FILE to directly write the response to disk.

Linus Kleen
  • 33,871
  • 11
  • 91
  • 99
  • Makes sense, but in this case I have retrieved the file from a different server using curl, so it is already in memory. – Joe Lencioni Jan 25 '11 at 19:45
  • @Joe I updated the answer. Another suggestion coming you way. – Linus Kleen Jan 25 '11 at 19:51
  • for the first solution you may have used [`stream_copy_to_stream`](http://php.net/stream_copy_to_stream) too ;-) – Arnaud Le Blanc Jan 25 '11 at 19:53
  • @user576875 Nah. Why go easy? :-) Just kidding. Thanks for the hint. – Linus Kleen Jan 25 '11 at 20:02
  • Cool idea using an output buffer, but doesn't the callback get called when the buffer is cleaned or flushed? Won't your code still hold the entire file in memory then? – Joe Lencioni Jan 25 '11 at 20:02
  • 6
    No need for the buffering. curl can write directly to a file using `curl_setopt('CURLOPT_FILE', ...)` – Marc B Jan 25 '11 at 20:05
  • @Marc I updated. Thank you. @Joe By setting `CURLOPT_BUFFERSIZE` to 512, cURL's internal buffer will be flushed each 512 bytes and read anew. Nothing's buffered, since the output buffer callback returns an empty string. – Linus Kleen Jan 25 '11 at 20:13
  • Oh, I see. Thanks for the clarification. This looks like it would work great! – Joe Lencioni Jan 25 '11 at 20:16
1

Writing line by line (or packet by packet in case of binary files) using functions like fwrite()

Mchl
  • 61,444
  • 9
  • 118
  • 120
0

Try this answer:

    $file   = fopen("file.json", "w");

    $pieces = str_split($content, 1024 * 4);
    foreach ($pieces as $piece) {
        fwrite($file, $piece, strlen($piece));
    }

    fclose($file);
Community
  • 1
  • 1
Eymen Elkum
  • 3,013
  • 1
  • 20
  • 34