How to handle large files with small process times

Question

On my webservice users are allowed to create forms and let theire friends or co-workers create data with them. The collected data can be downloaded as a zip file stream. Sometimes users have huge amounts of data (up to 2gb) and the server simply kills the php process for obvious reasons. Is it somehow possible to create such a file on client side without flash,java (btw java doesn't work anyway for most of my users) etc. ?

Is that 2GB in just one file or spread across multiple files? — Ja͢ck, Oct 12 '12 at 08:04
When is it killed, while downloading or while creating the file? And why is it killed? time, memory,....? — Nin, Oct 12 '12 at 08:08
the 2gb are multiple files, they get packed into a zip file. The zip file is downloaded while being created. And yeah its definitly a timeout. it kinda like zip_create(name.".zip"); foreach($files as $file) { addFile($file); flushNewStuff(); } zip_close(); — Alex, Oct 12 '12 at 08:32

score 0 · Answer 1 · answered Oct 12 '12 at 08:12

Increase your script timeout and memory usage.

use set_time_limit function Docs

And use ini_set for memory_limit parameter.

And one more solution is to give the clients file parts. i.e. give them a limit for downloading the number of records. i.e. 1-1000, 1001-2000 etc

score 0 · Answer 2 · edited May 23 '17 at 12:18

0

If you have control over the web server process I suggest you explore x-send-file as a solution to this.

See this SO question

In essence it will end the php process and send the file via the http server. This way time limits aren't an issue and you don't have a php instance hanging around.

edited May 23 '17 at 12:18

Community

1
1

answered Oct 12 '12 at 09:30

Luke

344
2
9

score 0 · Answer 3 · answered Oct 12 '12 at 10:43

Create a worker shell that keeps running in the background in a loop and checks for new data. If it finds new, unprocessed data have it prepare the download in the background. When the data is ready for download flag it as "ready" and inform the user (by email, polling via ajax for an status update, however you like) that his data was processed and is ready for download.

You can use nice to limit the CPU power used for that shell to avoid that it consumes all the available processing power and your site becomes slow.

That's exactly how I handle audio and video processing in one of my projects and it works fine.

How to handle large files with small process times

3 Answers3