How to POST a large amount of data within PHP curl without memory overhead?

Question

I'm currently using PHP curl extension to communicate with some HTTP APIs.

I use a bulk loader to perform a lot of operations at once. The bulk endpoint must be called with POST method so I use a code like :

<?php
$h = curl_init($url);
curl_setopt(CURLOPT_POST, true);
curl_setopt(CURLOPT_POSTFIELDS, $data);
curl_setopt(CURLOPT_RETURNTRANSFER, true);
$response = curl_exec($h);
curl_close($h);

The bulk endpoint allows me to send a huge amount of data (more than 200Mo at once). For the moment I need to load the data inside a variable which require PHP to be able to use enough memory...

I need to set memory_limit to a high value just for the bulk load...

Is there a way to use a file stream to send data with the curl PHP extension ? I saw the CURLOPT_INFILE and CURLOPT_READFUNCTION but it seems to don't work with POST method...

I also saw that the curl command line tool is able to perform a --data "@/path/to/file/content" which seems to be what I need...

Any ideas ?

Have you tried to deal with `CURLOPT_INFILE` ? I've just saw [this](http://stackoverflow.com/a/31696809/4469537) answer and I guess it works fine. — Sergey Chizhik, Sep 17 '15 at 10:13
I've already tried to play with the `CURLOPT_INFILE` but forgot to use the `CURLOPT_CUSTOMREQUEST` to hard set POST... I going to check it now ;) Thanks — shulard, Sep 17 '15 at 10:33

score 3 · Accepted Answer · answered Sep 17 '15 at 11:20

3

Use CURLOPT_INFILE

$curl = curl_init();
curl_setopt( $curl, CURLOPT_PUT, 1 );
curl_setopt( $curl, CURLOPT_INFILESIZE, filesize($tmpFile) );
curl_setopt( $curl, CURLOPT_INFILE, ($in=fopen($tmpFile, 'r')) );
curl_setopt( $curl, CURLOPT_CUSTOMREQUEST, 'POST' );
curl_setopt( $curl, CURLOPT_HTTPHEADER, [ 'Content-Type: application/json' ] );
curl_setopt( $curl, CURLOPT_URL, $url );
curl_setopt( $curl, CURLOPT_RETURNTRANSFER, 1 );
$result = curl_exec($curl);
curl_close($curl);
fclose($in);

answered Sep 17 '15 at 11:20

Ninju

2,522
2
15
21

Your answer just ended my hour-long struggle, thanks! Why on earth do I set the request method to `PUT`, then override it with `POST` again?! Who would assume uploads only happen via `PUT`..? – Moritz Friedrich Mar 20 '19 at 11:41
On my xampp server, if I omit this `PUT` line, $_FILES on the server side is empty... so it's better to do it the way above – Mikaël Mayer Jun 05 '21 at 12:48

How to POST a large amount of data within PHP curl without memory overhead?

1 Answers1

Linked