Downloading a large file (~500 Mb) to local storage from FTP, without breaking the server

Question

I'm required to download a large XML file from a remote FTP server to local storage so that I can process it.

I've defined an FTP driver that can access the file. However, because of the size of the file, PHP gives up allocating memory for the operation.

Storage::disk('ftp')->get('path/to/file/bigass.xml');

Is there any way that doesn't eat up memory and can download file without issues?

can you share how you implemented the get($filename) method? — Simonluca Landi, Sep 03 '18 at 06:03
give this a look https://stackoverflow.com/questions/597159/downloading-large-files-reliably-in-php — Maytham Fahmi, Sep 03 '18 at 06:04
@SimonlucaLandi - it's the regular Laravel `get` method. No customisation. — TheBigK, Sep 03 '18 at 06:39
please have a look at https://stackoverflow.com/questions/19509440/increase-the-limit-of-file-download-size-in-document-management-system — manu, Sep 03 '18 at 06:53
try this lib which supports streams i believe https://flysystem.thephpleague.com/docs/adapter/sftp/ — delboy1978uk, Sep 03 '18 at 08:40
also check this answer which uses flysystem https://stackoverflow.com/questions/26020515/phpleague-flysystem-read-and-write-to-large-file-on-server — delboy1978uk, Sep 03 '18 at 08:41

score 0 · Answer 1 · answered Sep 03 '18 at 07:05

0

I suggest to switch to a "plain old" curl solution, using something like this:

$curl = curl_init();
$fh   = fopen("localfilename.xml", 'w');
curl_setopt($curl, CURLOPT_URL, "ftp://{$ftp_username}:{$ftp_password}@{$ftp_server}/path/to/file/bigass.xml");
curl_setopt($curl, CURLOPT_RETURNTRANSFER, 1);
$result = curl_exec($curl);
fwrite($fh, $result);
fclose($fh);
curl_close($curl);

answered Sep 03 '18 at 07:05

Simonluca Landi

931
8
21

How does it work differently? I mean, Does curl support streaming? – TheBigK Sep 03 '18 at 07:10

Downloading a large file (~500 Mb) to local storage from FTP, without breaking the server

1 Answers1