4GB HTTP File Uploads Using jQuery-File-Upload, Apache and PHP

Question

This problem seems directly related to the infamous 2GB limit and I'm not sure at this atge if its a 32bit PHP issue. I've seen the comments related to HTTP not being designed for such large files. However, I would rather exhaust this direction before deciding on any fundamental change to what has been proven to work so far. As the title suggest I need to upload >2GB files, largest so far 3.8GB. I have a form built using jQuery-file-upload which has had no problems sending any files (all less <2GB) until these large ones started. The 3.8GB one takes maybe 5 minutes to upload from File-Upload and the progress bar completes to 100%. However the reported error after upload is a type 1 which suggests the file size limit has been breached.

If I monitor the disk usage on the server during upload it will have the following sample properties: Before upload the disk is 30% used, as the upload continues this usage increases, 31%,32%....45%,46%etc. However at around the 2GB point the disk usage drops back to 30% while the client/browser File-Upload continues. When the servers disk usage stops increasing the client may only be at 60% of the upload. Yet the client still continues to 100% but isn't being accepted by the server as the disk status never moves off 30%.

I'm running 64Bit Ubuntu (latest) with Apache/2.2.22 (Ubuntu) and PHP Version 5.3.10-1ubuntu3.2 After many days on and off searching for a solution I still cannot get the 3.8GB file to upload after changing so many settings. I'll list the sort of changes below but at this stage I think it may be a 32bit PHP issue so if anyone can suggest a link worth following or solution worth trying I'd appreciate it.

In Apache2 I've set the following:

- apache2.conf I've set Timeout to 900
- httpd.conf I've set LimitRequestBody to 0
- .htaccess in the file upload directory I've set:
    - LimitRequestBody to 4939212390
    - php_value upload_max_filesize 4831838208
    - php_value post_max_size 4939212390

In php.ini I've set the following:

- UPLOAD_MAX_FILESIZE 4831838208
- POST_MAX_SIZE 4939212390
- max_execution_time 120
- max_input_time 60
- memory_limit 128M

If I run the following on the server it would seem the PHP does not have a 32bit issue but I'm not sure at this stage.

php -r "echo PHP_INT_MAX;"
9223372036854775807

As I mentioned any ideas would be greatly appreciated.

UPDATE:

Have solved this issue, so thanks to @BogdanBurim for suggesting the back to basics approach:

I have managed to upload a 3.8 GB file over HTTP with the following settings:

In Apache2 I've set the following:

- apache2.conf I've set Timeout to 900
- httpd.conf I've set LimitRequestBody to 0
- .htaccess in the file upload directory I've set:
    - LimitRequestBody to 0
    - php_value upload_max_filesize 0
    - php_value post_max_size 4939212390
- .htaccess in the php temp directory (in my case its /tmp/) I've set:
    - LimitRequestBody to 0
    - php_value upload_max_filesize 0
    - php_value post_max_size 4939212390

In php.ini I've set the following:

- UPLOAD_MAX_FILESIZE 0
- POST_MAX_SIZE 4939212390
- max_execution_time 120
- max_input_time 60
- memory_limit 128M

The only other important part of this solution was having to remove the MAX_FILE_SIZE, HTML from the upload form:

<input type="hidden" name="MAX_FILE_SIZE" value="4939212390" />

Having this set continually caused a PHP type 2 error, so php couldn't handle the greater than 32bit integer that is set. Removing it caused PHP type 1 errors until I changed the UPLOAD_MAX_FILESIZE to 0 everywhere and hey presto it now works!!!!

http://php.net/manual/en/features.file-upload.errors.php

related: http://stackoverflow.com/questions/5053290/large-file-upload-though-html-form-more-than-2-gb — valentinas, Nov 26 '12 at 23:00
Try to upload file with simple HTML form and .. Will it break down? You perhaps will see the exact error message. — Bogdan Burym, Nov 27 '12 at 09:55

score 6 · Accepted Answer · answered Nov 29 '12 at 16:39

Have solved this issue, so thanks to @BogdanBurim for suggesting the back to basics approach:

I have managed to upload a 3.8 GB file over HTTP with the following settings:

In Apache2 I've set the following:

- apache2.conf I've set Timeout to 900
- httpd.conf I've set LimitRequestBody to 0
- .htaccess in the file upload directory I've set:
    - LimitRequestBody to 0
    - php_value upload_max_filesize 0
    - php_value post_max_size 4939212390
- .htaccess in the php temp directory (in my case its /tmp/) I've set:
    - LimitRequestBody to 0
    - php_value upload_max_filesize 0
    - php_value post_max_size 4939212390

In php.ini I've set the following:

- UPLOAD_MAX_FILESIZE 0
- POST_MAX_SIZE 4939212390
- max_execution_time 120
- max_input_time 60
- memory_limit 128M

The only other important part of this solution was having to remove the MAX_FILE_SIZE, HTML from the upload form:

<input type="hidden" name="MAX_FILE_SIZE" value="4939212390" />

Having this set continually caused a PHP type 2 error, so php couldn't handle the greater than 32bit integer that is set. Removing it caused PHP type 1 errors until I changed the UPLOAD_MAX_FILESIZE to 0 everywhere and hey presto it now works!!!!

http://php.net/manual/en/features.file-upload.errors.php

Did you test in all Browser versions (IE 7+ and FF12+ ) and does it work in all of these browsers? Does using simple `HTML` ` — Vikas V, Dec 05 '12 at 09:52
@VikasV I didn't test it on any IE platforms yet but will do now that you ask. It is working fine on FF and CH, however the problem was never the client browser for me. In both FF and CH if sent the file but always reported a partial file upload problem for 2GB+ files at the server end, which has been fixed for me as per the above settings. — Diplonics, Dec 09 '12 at 22:05
LimitRequestBody defaults to `0` (unlimited) in [Apache 2.4](https://httpd.apache.org/docs/2.4/mod/core.html#limitrequestbody), perhaps other versions as well. — chb, Oct 08 '17 at 15:39

score 0 · Answer 2 · answered Nov 05 '13 at 09:52

If you have access to the apache virtualhost config you can also change these settings for a specific upload URL (you can also add this to your .htaccess file):

With this code:

<LocationMatch "/index.php/url-of-your-upload.php">
    php_value max_execution_time 0
    php_value upload_max_filesize 0
    php_value post_max_size 4939212390
    php_value memory_limit 4G
    LimitRequestBody 0
</LocationMatch>

The LocationMatch directives allow you to choose the url (you can use reg exp)

This gave me a 500 error when placing it on my .htaccess file. — Tomas Gonzalez, Apr 13 '15 at 02:39

score -7 · Answer 3 · answered Nov 26 '12 at 23:04

-7

The question is why would you use your browser to upload > 1GB. Consider top file sharing services most limit file upload size to 1GB via browser. What happens if user-download fails you would have to restart the whole process.

Have you looked in to alternatives if you have such large file uploads, torrents, ftp or personal upload client like rapidshare, fileupload, megaupload etc. Have.

You are limited because of the file size limit set to POST's. If you are determined to use HTTP to upload bigger files which i would discourage, the optimal solution would be spliting the file and uploading the smaller splited parts then reassembling the file.

answered Nov 26 '12 at 23:04

Sterling Duchess

1,970
16
51
91

2

"What happens if user-download fails you would have to restart the whole process." emmm, no? http://www.w3.org/Protocols/rfc2616/rfc2616-sec14.html#sec14.16 – valentinas Nov 26 '12 at 23:53
Yes that works very well ending up in corrupt files. It's stupid to upload such large files this way. – Sterling Duchess Nov 27 '12 at 01:25
Thanks @kellax but the point of the question is to exhaust information on the issue. It also has nothing to with downloads. If you can find hard evidence of POST limits please link to such as I cannot find any, in either a client or server context. Also the fact that jQuery-File-Upload has no problem handling a 3.8GB file in both a Chrome or Firefox browser it would seem there is no browser issue. (Firefox crashes on such a large file when firebug is running but is fine otherwise, as an aside). Lastly, being stupid is irrelevant, the process has worked fine until this >2GB requirement. – Diplonics Nov 27 '12 at 08:57

4GB HTTP File Uploads Using jQuery-File-Upload, Apache and PHP

3 Answers3

Linked