Curl hangs when downloading large file (500MB+)

Question

I'm using cURL to download large XML files (between 500MB and 1GB) from a remote server. Although the script works fine for smaller test files, every time I try to download a file larger than a few hundred megabytes, the script seems to hang - it doesn't quit, there's no error message, it just hangs there.

I'm executing the script from the command line (CLI), so PHP itself should not time out. I have also tried cURL's verbose mode, but this shows nothing beyond the initial connection. Every time I download the file, it stops at exactly the same size (463.3MB). The file's XML at this point is incomplete.

Any ideas much appreciated.

        $ch = curl_init();
    $fh = fopen($filename, 'w');

    curl_setopt($ch,CURLOPT_URL, $url);
    curl_setopt($ch, CURLOPT_FOLLOWLOCATION, TRUE );
    curl_setopt($ch, CURLOPT_FILE, $fh);
    curl_setopt($ch, CURLOPT_HEADER, 0);
    curl_setopt($ch, CURLOPT_TIMEOUT, 0);

    if(curl_exec($ch) === false)
    {
        echo 'Curl error: ' . curl_error($ch) . "\n";
    }
    else
    {
        echo 'Operation completed without any errors';
    }

    $response = array(
        'header' => curl_getinfo($ch)
    );

    curl_close($ch);
    fclose($fh);

    if($response['header']['http_code'] == 200) {
        echo "File downloaded and saved as " . $filename . "\n";
    }

Again, this script works fine with smaller files, but with the large file I try to download it does not even get as far as printing out an error message.

Could this be something else (Ubuntu 10.04 on Linode) terminating the script? As far as I understand, my webserver shouldn't matter here since I am running it through CLI.

Thanks,

Matt

Try setting this option to false: CURLOPT_NOPROGRESS It enables a progress bar, so you can see if you are still actually downloading the file. — psion, Jan 13 '12 at 08:32

score 1 · Answer 1 · answered Nov 02 '11 at 15:50

The files appear to be complete when they stop downloading right? Adding -m 10800 to the command will timeout and end the transfer after the specified number of seconds. This will work if you set the timeout to be longer than the transfer takes, but is still annoying.

score 0 · Answer 2 · edited Jun 20 '12 at 17:14

0

You might be out of disk space on the partition you are saving, Or running over quota for the user running the script.

edited Jun 20 '12 at 17:14

Jon Lin

142,182
29
220
220

answered Jun 20 '12 at 17:08

Rodrigo O

1

score 0 · Accepted Answer · edited May 23 '17 at 10:30

0

Check this post, maybe you can try to download file as parts. Or if you have access to remote server you can try to archive file and then download it. You can check your php.ini configuration too. See for file size, memory limits and other.

edited May 23 '17 at 10:30

Community

1
1

answered Mar 27 '11 at 23:52

Faraona

1,685
2
22
33

Thanks but the RANGE option doesn't work for the server I am downloading from (which I have no control over). Also can't see anything ominous in PHP.ini - any specific suggestions? – Mateo Mar 28 '11 at 00:06
Why don't use direct curl from terminal, not from php script? – Faraona Mar 28 '11 at 00:14

Curl hangs when downloading large file (500MB+)

3 Answers3