0

I'm working on an upload tool to upload large files (1GB - 4GB) to a FTP server. I used HTML5 to slice the uploaded file into 1MB chunks and then upload each of those chunks to a temp folder. As soon as every chunk is uploaded a PHP script creates the file out of those chunks:

if (($fp = fopen(UPLOAD_DIR.$fileName, 'w')) !== false) {
    for ($i = 1; $i <= $totalFiles; $i++) {
        fwrite($fp, file_get_contents($tempDir.'/'.$fileName.'.part'.$i));               
    }
    fclose($fp);        
}

Everything works for smaller file sizes (tested with files around 1GB - 1.5GB), but as soon as i upload very large files (e.g. 3GB) my PHP script stops with a 504 Gateway timeout. The 'max_execution_time' in my php.ini is set to 90. Is there any way to avoid the 504 timeout in this case or a way to speed up the script, which creates the file? I could try to change 'max_execution_time', but that doesn't seem to be the perfect solution. Any suggestions?

EDIT: The script is running on a Linux System

mrksbnch
  • 1,792
  • 2
  • 28
  • 46

1 Answers1

1

I don't know what operating system your server is running, but if it is linux, you could try it directly from the command line using for example exec.

Something like:

exec('cat ' . $tempDir . '/' . $fileName . '.part* > ' . UPLOAD_DIR . $fileName);

You might need to use escapeshellcmd() on your variables if you don't control them.

You can also generate a list of partial file names in a loop if .part* does not give you the correct order.

jeroen
  • 91,079
  • 21
  • 114
  • 132
  • Thanks, it's Linux, i'll give it a try. Haven't worked with shell commands yet though. But hopefully i'll figure it out. Does exec delete the .part* files after it's done? – mrksbnch Mar 05 '14 at 14:36
  • @demrks If the example above does not generate the file in the correct order, just use your current loop to generate a space-separated string of file-names in the right order. – jeroen Mar 05 '14 at 14:40
  • What parts of the exec commands do i have to change then if i generate a long string like `$fileNames = 'fileName.part1 fileName.part2 fileName.part3'` – mrksbnch Mar 05 '14 at 14:48
  • 1
    @demrks You would need to generate complete file-names, including the `$tempDir` path and then the command would be `'cat ' . $fileNames . ' > ' . UPLOAD_DIR . $fileName` – jeroen Mar 05 '14 at 14:51
  • Is there any way to do anything as soon as the `exec` command was successful? I need to delete the 'temp' folder and store the file name in a database after the file was created by the shell command. Thanks. – mrksbnch Mar 05 '14 at 18:11
  • @demrks There is nothing asynchronous there so you can just put it after. – jeroen Mar 05 '14 at 18:29
  • Ok, even better. But if php has to wait until the shell command was executed (which might take longer than the `max_execution_time` in my case), I'm wondering why this exec doesn't affect `max_execution_time` then? – mrksbnch Mar 05 '14 at 18:54
  • 1
    @demrks It does, but I think native file system operations are a lot faster than doing the same thing through php. – jeroen Mar 05 '14 at 21:03
  • `exec` seems to fail for large files: a 1.8GB file worked, but a 2.88GB file didn't work. The command looked like `cat '../uploads/temp/2878443218-filezip/file.zip.part1' ... '../uploads/temp/2878443218-filezip/file.zip.part2745' > '../uploads/file.zip'`. The error return code of `exec` was [127](http://stackoverflow.com/questions/1763156/127-return-code-from) for the large file. Is it possible that there is a file limit in `cat`? – mrksbnch Mar 06 '14 at 13:15