Serving large files with PHP

Question

So I am trying to serve large files via a PHP script, they are not in a web accessible directory, so this is the best way I can figure to provide access to them.

The only way I could think of off the bat to serve this file is by loading it into memory (fopen, fread, ect.), setting the header data to the proper MIME type, and then just echoing the entire contents of the file.

The problem with this is, I have to load these ~700MB files into memory all at once, and keep the entire thing there till the download is finished. It would be nice if I could stream in the parts that I need as they are downloading.

Any ideas?

Answer 1

You don't need to read the whole thing - just enter a loop reading it in, say, 32Kb chunks and sending it as output. Better yet, use fpassthru which does much the same thing for you....

$name = 'mybigfile.zip';
$fp = fopen($name, 'rb');

// send the right headers
header("Content-Type: application/zip");
header("Content-Length: " . filesize($name));

// dump the file and stop the script
fpassthru($fp);
exit;

even less lines if you use readfile, which doesn't need the fopen call...

$name = 'mybigfile.zip';

// send the right headers
header("Content-Type: application/zip");
header("Content-Length: " . filesize($name));

// dump the file and stop the script
readfile($name);
exit;

If you want to get even cuter, you can support the Content-Range header which lets clients request a particular byte range of your file. This is particularly useful for serving PDF files to Adobe Acrobat, which just requests the chunks of the file it needs to render the current page. It's a bit involved, but see this for an example.

Answer 2

The best way to send big files with php is the X-Sendfile header. It allows the webserver to serve files much faster through zero-copy mechanisms like sendfile(2). It is supported by lighttpd and apache with a plugin.

Example:

$file = "/absolute/path/to/file"; // can be protected by .htaccess
header('X-Sendfile: '.$file);
header('Content-type: application/octet-stream');
header('Content-Disposition: attachment; filename="'.basename($file).'"');
// other headers ...
exit;

The server reads the X-Sendfile header and sends out the file.

Answer 3

While fpassthru() has been my first choice in the past, the PHP manual actually recommends* using readfile() instead, if you are just dumping the file as-is to the client.

* "If you just want to dump the contents of a file to the output buffer, without first modifying it or seeking to a particular offset, you may want to use the readfile(), which saves you the fopen() call." —PHP manual

Answer 4

If your files are not accessible by the web server because the path is not in your web serving directory (htdocs) then you can make a symbolic link (symlink) to that folder in your web serving directory to avoid passing all traffic trough php.

You can do something like this

ln -s /home/files/big_files_folder /home/www/htdocs

Using php for serving static files is a lot slower, if you have high traffic, memory consumption will be very large and it may not handle a big number of requests.

Answer 5

Have a look at fpassthru(). In more recent versions of PHP this should serve the files without keeping them in memory, as this comment states.

Answer 6

Strange, neither fpassthru() nor readfile() did it for me, always had a memory error. I resorted to use passthru() without the 'f':

$name = 'mybigfile.zip';
// send the right headers
header("Content-Type: application/zip");
header("Content-Length: " . filesize($name));
// dump the file and stop the script
passthru('/bin/cat '.$filename);
exit;

this execs 'cat' Unix command and send its output to the browser.

comment for slim: the reason you just don't put a symlink to somewhere is webspace is SECURITY.

Answer 7

One of benefits of fpassthru() is that this function can work not only with files but any valid handle. Socket for example.

And readfile() must be a little faster, cause of using OS caching mechanism, if possible (as like as file_get_contents()).

One more tip. fpassthru() hold handle open, until client gets content (which may require quite a long time on slow connect), and so you must use some locking mechanism if parallel writes to this file possible.

Answer 8

The Python answers are all good. But is there any reason you can't make a web accessible directory containing symbolic links to the actual files? It may take some extra server configuration, but it ought to work.

Answer 9

If you want to do it right, PHP alone can't do it. You would want to serve the file by using Nginx's X-Accel-Redirect (Recommended) or Apache's X-Sendfile, which are built exactly for this purpose.

I will include in this answer some text found on this article.

Why not serve the files with PHP:

• Done naively, the file is read into memory and then served. If the files are large, this could cause your server to run out of memory.
• Caching headers are often not set correctly. This causes web browsers to re-download the file multiple times even if it hasn't changed.
• Support for HEAD requests and range requests is typically not automatically supported. If the files are large, serving such files ties up a worker process or thread. This can lead to starvation if there are limited workers available. Increasing the number of workers can cause your server to run out of memory.

NGINX handles all of these things properly. So let's handle permission checks in the application and let NGINX serve the actual file. This is where internal redirects come in. The idea is simple: you can configure a location entry as usual when serving regular files.

Add this to your nginx server block:

location /protected_files/ {
    internal;
    alias /var/www/my_folder_with_protected_files/;
}

In your project, require the HTTP Foundation package:

composer require symfony/http-foundation

Serve the files in PHP using Nginx:

use Symfony\Component\HttpFoundation\BinaryFileResponse;

$real_path = '/var/www/my_folder_with_protected_files/foo.pdf';
$x_accel_redirect_path = '/protected_files/foo.pdf';

BinaryFileResponse::trustXSendfileTypeHeader();
$response = new BinaryFileResponse( $real_path );
$response->headers->set( 'X-Accel-Redirect', $accel_file );
$response->sendHeaders();
exit;

This should be the basic you need to get started.

Here's a more complete example serving an Inline PDF:

use Symfony\Component\HttpFoundation\BinaryFileResponse;
use Symfony\Component\HttpFoundation\File\File;
use Symfony\Component\HttpFoundation\ResponseHeaderBag;

$real_path = '/var/www/my_folder_with_protected_files/foo.pdf';
$x_accel_redirect_path = '/protected_files/foo.pdf';

$file = new File( $file_path );

BinaryFileResponse::trustXSendfileTypeHeader();
$response = new BinaryFileResponse( $file_path );
$response->setImmutable( true );
$response->setPublic();
$response->setAutoEtag();
$response->setAutoLastModified();
$response->headers->set( 'Content-Type', 'application/pdf' );
$response->headers->set( 'Content-Length', $file->getSize() );
$response->headers->set( 'X-Sendfile-Type', 'X-Accel-Redirect' );
$response->headers->set( 'X-Accel-Redirect', $accel_file );
$response->headers->set( 'X-Accel-Expires', 60 * 60 * 24 * 90 ); // 90 days
$response->headers->set( 'X-Accel-Limit-Rate', 10485760 ); // 10mb/s
$response->headers->set( 'X-Accel-Buffering', 'yes' );
$response->setContentDisposition( ResponseHeaderBag::DISPOSITION_INLINE, basename( $file_path ) ); // view in browser. Change to DISPOSITION_ATTACHMENT to download
$response->sendHeaders();
exit;