How to force download of big files without using too much memory?

Question

I'm trying to serve large zip files to users. When there are 2 concurrent connections, the server runs out of memory (RAM). I increased the amount of memory from 300MB to 4GB (Dreamhost VPS) and then it worked fine.

I need to allow a lot more than 2 concurrent connections. The actual 4GB would allow something like 20 concurrent connections (too bad).

Well, the current code I'm using, needs the double of memory then the actual file size. That's too bad. I want something like "streaming" the file to user. So I would allocate not more than the chunk being served to users.

The following code is the one I'm using in CodeIgniter (PHP framework):

ini_set('memory_limit', '300M'); // it was the maximum amount of memory from my server
set_time_limit(0); // to avoid the connection being terminated by the server when serving bad connection downloads
force_download("download.zip", file_get_contents("../downloads/big_file_80M.zip"));exit;

The force_download function is as follows (CodeIgniter default helper function):

function force_download($filename = '', $data = '')
{
    if ($filename == '' OR $data == '')
    {
        return FALSE;
    }

    // Try to determine if the filename includes a file extension.
    // We need it in order to set the MIME type
    if (FALSE === strpos($filename, '.'))
    {
        return FALSE;
    }

    // Grab the file extension
    $x = explode('.', $filename);
    $extension = end($x);

    // Load the mime types
    @include(APPPATH.'config/mimes'.EXT);

    // Set a default mime if we can't find it
    if ( ! isset($mimes[$extension]))
    {
        $mime = 'application/octet-stream';
    }
    else
    {
        $mime = (is_array($mimes[$extension])) ? $mimes[$extension][0] : $mimes[$extension];
    }

    // Generate the server headers
    if (strpos($_SERVER['HTTP_USER_AGENT'], "MSIE") !== FALSE)
    {
        header('Content-Type: "'.$mime.'"');
        header('Content-Disposition: attachment; filename="'.$filename.'"');
        header('Expires: 0');
        header('Cache-Control: must-revalidate, post-check=0, pre-check=0');
        header("Content-Transfer-Encoding: binary");
        header('Pragma: public');
        header("Content-Length: ".strlen($data));
    }
    else
    {
        header('Content-Type: "'.$mime.'"');
        header('Content-Disposition: attachment; filename="'.$filename.'"');
        header("Content-Transfer-Encoding: binary");
        header('Expires: 0');
        header('Pragma: no-cache');
        header("Content-Length: ".strlen($data));
    }

    exit($data);
}

I tried some chunk based codes that I found in Google, but the file always was delivered corrupted. Probably because of bad code.

Could anyone help me?

Have you tried to redirect to a file using header `Location`? — Daniel, Jun 01 '11 at 02:27
Sounds to me like you are much better off just giving the users a direct link to the files... — NotMe, Jun 01 '11 at 02:35
I forgot to tell that the files are in a folder not accessible through web. This is for security reasons. I just serve the file if the user pass in a authentication process. I'll try the suggestions below and will come back to vote for the best answer. — Leandro Alves, Jun 01 '11 at 17:54
Thanks for the text revision, @p.campbell I think I was too tired last night... :) — Leandro Alves, Jun 01 '11 at 19:28

score 3 · Accepted Answer · edited May 23 '17 at 12:31

3

There are some ideas over in this thread. I don't know if the readfile() method will save memory, but it sounds promising.

edited May 23 '17 at 12:31

Community

1
1

answered Jun 01 '11 at 02:33

King Skippus

3,801
1
24
24

yes, `readfile` does save memory, because every chunk of the file it reads gets outputted directly to the browser without storing it in a variable (and so without using extra memory, just the needed for the file chunk). – Carlos Campderrós Jun 01 '11 at 13:13
1

That worked just fine. I just added some extra headers, so the iPhone doesn't alert an error with the download. `header('Content-Disposition: attachment; filename="download.zip"'); header('Expires: 0'); header('Cache-Control: must-revalidate, post-check=0, pre-check=0'); header("Content-Transfer-Encoding: binary"); header('Pragma: public'); header("Content-Length: ".filesize($filename)); readfile($filename);exit;` – Leandro Alves Jun 01 '11 at 19:23
2

I originally thought that `readfile` or `fpassthru` would work as well, but just ran into an issue today where it appears that `readfile` actually still reads the entire file into memory. Perhaps that's been changed in a newer version of PHP though (I was using 5.2). – Eric Petroelje Jun 03 '11 at 14:41

score 3 · Answer 2 · answered Jun 01 '11 at 12:38

You're sending the contents ($data) of this file via PHP?

If so, each Apache process handling this will end up growing to the size of this file, as that data will be cached.

Your ONLY solution is to not send file contents/data via PHP and simply redirect the user to a download URL on the filesystem.

Use a generated and unique symlink, or a hidden location.

score 1 · Answer 3 · answered Dec 22 '19 at 22:43

I searched too many scripts and advises and nothing worked for my 400MB PDF file. I ended up using mod_rewrite and here's the solution it works great https://unix.stackexchange.com/questions/88874/control-apache-referrer-to-restrict-downloads-in-htaccess-file The code only allows download from a referrer you specifies and forbids direct download

 RewriteEngine On
RewriteCond %{HTTP_REFERER} !^http://yourdomain.com/.* [NC]
RewriteRule .* - [F]

score 0 · Answer 4 · answered Jun 01 '11 at 02:38

0

You can't use $data with whole file data inside it. Try pass to this function not the content of file only it's path. Next send all headers once and after that read part of this file using fread(), echo that chunk, call flush() and repeat. If any other header will be send in the meantime then finally transfer will be corrupted.

answered Jun 01 '11 at 02:38

Arek Jablonski

349
1
7

`readfile` reads whole file at once, in my proposition I used `fread` (can be used `fgets` as well) because if the file will be read in i'e 1MB chunks then that memory can be freed when next chunk will be assigned to the same variable. – Arek Jablonski Jun 01 '11 at 15:58
1

sorry but no, `readfile` reads the file in 8K chunks. I've actually dug through php code because that intrigued me, and confirmed it. [readfile source, line 1383](http://svn.php.net/viewvc/php/php-src/trunk/ext/standard/file.c?revision=311543&view=markup), `php_stream_passthru` in there is DEFINED to `_php_stream_passthru` [(source, line 453)](http://svn.php.net/viewvc/php/php-src/trunk/main/php_streams.h?revision=311422&view=markup) and that later function is in [this file, line 1314](http://svn.php.net/viewvc/php/php-src/trunk/main/streams/streams.c?revision=311545&view=markup) – Carlos Campderrós Jun 02 '11 at 09:45
In this case it should be described in the php documentation. – Arek Jablonski Jun 02 '11 at 10:51

score 0 · Answer 5 · answered Jun 01 '11 at 02:42

0

Symlink the big file to your document root (assuming its not an authorized only file), then let Apache handle it. (That way you can accept byte ranges as well)

answered Jun 01 '11 at 02:42

Petah

45,477
28
157
213

Presumes that Apache has FollowSymLinks enabled and that the OP isn't using the PHP download script to add a security layer, but otherwise a good idea. – Marc B Jun 01 '11 at 03:59

score 0 · Answer 6 · edited Nov 17 '12 at 19:25

0

Add your ini_set before SESSION_START();

edited Nov 17 '12 at 19:25

Zaheer Ahmed

28,160
11
74
110

answered Jul 07 '11 at 11:50

Emile Swarts

1

How to force download of big files without using too much memory?

6 Answers6

Linked