Why does readfile() exhaust PHP memory?

Question

I've seen many questions about how to efficiently use PHP to download files rather than allowing direct HTTP requests (to keep files secure, to track downloads, etc.).

The answer is almost always PHP readfile().

BUT, although it works great during testing with huge files, when it's on a live site with hundreds of users, downloads start to hang and PHP memory limits are exhausted.

So what is it about how readfile() works that causes memory to blow up so bad when traffic is high? I thought it's supposed to bypass heavy use of PHP memory by writing directly to the output buffer?

EDIT: (To clarify, I'm looking for a "why", not "what can I do". I think that Apache's mod_xsendfile is the best way to circumvent)

Do you have PHP output buffering on (via `ob_start()`)? If so, PHP's going to catch the file contents and cause memory issues. — ceejayoz, Jul 08 '11 at 17:27
Using the example on the linked PHP doc, I don't -- I have ob_clean(); flush(); readfile($file); — tmsimont, Jul 08 '11 at 17:29

score 6 · Accepted Answer · answered Jul 08 '11 at 17:28

6

Description
int readfile ( string $filename [, bool $use_include_path = false [, resource $context ]] )
Reads a file and writes it to the output buffer*.

PHP has to read the file and it writes to the output buffer. So, for 300Mb file, no matter what the implementation you wrote (by many small segments, or by 1 big chunk) PHP has to read through 300Mb of file eventually.

If multiple user has to download the file, there will be a problem. (In one server, hosting providers will limit memory given to each hosting user. With such limited memory, using buffer is not going to be a good idea. )

I think using the direct link to download a file is a much better approach for big files.

answered Jul 08 '11 at 17:28

user482594

16,878
21
72
108

So if PHP is "writing to the output buffer" for UserA -- it's using memory that is then unavailable to UserB and UserC? Leading to an eventual overuse of memory? – tmsimont Jul 08 '11 at 17:33
Yes. Since PHP does not know when the other user is gonna download the same file, PHP will just read the file to buffer and write them to a requester as soon as possible... If 100 users request the same 10MB file, the memory usage will likely be 1000MB. – user482594 Jul 08 '11 at 17:49
makes sense to me... so readfile() doesn't quite do the trick, and should be avoided in high traffic situations. thanks. – tmsimont Jul 08 '11 at 17:53
your answer indicates this problem exists even when buffering small segments. In that particular case, your 100 users requesting 10MB file would NOT result in 1000MB of memory usage, but 100 x buffer size. Correct? – horatio Jul 08 '11 at 18:29
Yeah that is right, I was saying that the 'total' memory usage used up will be 1000MB at the end. At certain moment, it will be `100 * buffer_size` – user482594 Jul 08 '11 at 18:44

score 3 · Answer 2 · answered Nov 16 '12 at 12:57

3

If you have output buffering on than use ob_end_flush() right before the call to readfile()

header(...);
ob_end_flush();
@readfile($file);

answered Nov 16 '12 at 12:57

dgpro

347
3
10

I tried adding this before doing readfile on a 2GB file, but it still went over the memory limit – NDM Mar 29 '13 at 15:41
As mentioned above, output buffering is not the problem. – reggie Jun 09 '13 at 11:56

score 3 · Answer 3 · edited May 23 '17 at 11:47

3

As mentioned here: "Allowed memory .. exhausted" when using readfile, the following block of code at the top of the php file did the trick for me.

This will checks if php output buffering is active. If so it turns it off.

if (ob_get_level()) {
    ob_end_clean();
}

edited May 23 '17 at 11:47

Community

1
1

answered Aug 17 '16 at 01:26

Austin Ginder

516
3
5

Since the question is "why", I think you should provide an explanation rather than a fix. After all, you're basically copying an existing answer. – Álvaro González Aug 17 '16 at 07:24
Ah yes. I went ahead and added a short explanation. – Austin Ginder Aug 18 '16 at 02:48

score 1 · Answer 4 · answered Oct 05 '12 at 15:30

You might want to turn off output buffering altogether for that particular location, using PHP's output_buffering configuration directive.

Apache example:

<Directory "/your/downloadable/files">
     ...
     php_admin_value output_buffering "0"
     ...
</Directory>

"Off" as the value seems to work as well, while it really should throw an error. At least according to how other types are converted to booleans in PHP. *shrugs*

score 1 · Answer 5 · answered Jul 25 '13 at 14:03

Came up with this idea in the past (as part of my library) to avoid high memory usage:

function suTunnelStream( $sUrl, $sMimeType, $sCharType = null )
{
  $f = @fopen( $sUrl, 'rb' );
  if( $f === false )
   { return false; }

  $b = false;
  $u = true;

  while( $u !== false && !feof($f ))
  { 
    $u = @fread( $f, 1024 );
    if( $u !== false )
    {  
      if( !$b )
       { $b = true;
         suClearOutputBuffers();
         suCachedHeader( 0, $sMimeType, $sCharType, null, !suIsValidString($sCharType)?('content-disposition: attachment; filename="'.suUniqueId($sUrl).'"'):null );
       } 
      echo $u; 
    }
 } 

  @fclose( $f );
  return ( $b && $u !== false );
}

Maybe this can give you some inspiration.

score 0 · Answer 6 · answered Jul 08 '11 at 17:28

Well, it is memory intensive function. I would pipe users to a static server that has specific rule set in place to control downloads instead of using readfile().

If that's not an option add more RAM to satisfy the load or introduce queuing system that gracefully controls server usage.

Why does readfile() exhaust PHP memory?

6 Answers6

Linked