1

I use this cmd to successfully download result.csv

curl.exe https://example.com/rest -u "xx:yy" -F xmlRequest=@file.xml > result.csv

I want to do this using PHP. But no avail... Help please.

This is my PHP:

$url = 'https://example.com/rest';
$xml = '<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<request xmlns="http://example.com"><search path="report_date" value="2023-07-01T00:00:00+0000"/>'; //and so on..
$ch = curl_init();
curl_setopt($ch, CURLOPT_USERPWD, 'xx:yy');
curl_setopt($ch, CURLOPT_URL,$url);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
curl_setopt($ch, CURLOPT_TIMEOUT, 10);
curl_setopt($ch, CURLOPT_POST, true);
curl_setopt($ch, CURLOPT_POSTFIELDS, "xmlRequest=$xml");
curl_setopt($ch, CURLOPT_HTTPHEADER, array('Content-Type: text/xml'));
curl_setopt($ch, CURLOPT_FOLLOWLOCATION, true);
curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, false);
curl_setopt($ch, CURLOPT_SSL_VERIFYHOST, false);

$data = curl_exec($ch);
var_dump($data); //just to check if it has requested data (will save to file later)
if(curl_errno($ch))
    print curl_error($ch);
else
    curl_close($ch);

It does not have requested data. But it gives this

string(346) " 500 An error has occured. Please try again and if the problem persists, contact us. "

Brian Tompsett - 汤莱恩
  • 5,753
  • 72
  • 57
  • 129
Pekzz
  • 45
  • 6
  • The `@file.xml` syntax causes cURL to perform an actual file upload, that is something different, than just providing a normal text parameter containing the XML in string form. And with a file upload, the original command did not send `Content-Type: text/xml` either, but `Content-Type: multipart/form-data`. – CBroe Jul 12 '23 at 06:09
  • If you want to upload an actually existing file, use the `CURLFile` class. If you want to "fake" a file upload using data you got in string form, use the `CURLStringFile` class. – CBroe Jul 12 '23 at 06:12
  • I changed into `$xml = '@file.xml';` and `Content-Type: multipart/form-data` - but still error 500 result. I'll try the other one now. Thanks – Pekzz Jul 12 '23 at 06:37
  • Now I changed `$xml['file'] = new CurlFile('file.xml', 'multipart/form-data', 'file.xml');` But it gives Warning: Array to string conversion in `curl_setopt($ch, CURLOPT_POSTFIELDS, "xmlRequest=$xml");` What should I do from here? – Pekzz Jul 12 '23 at 07:12
  • `xmlRequest=$xml` is still totally wrong for a multipart request. You should provide an array to CURLOPT_POSTFIELDS, and let cURL handle the rest (including setting the Content-Type header, because that needs to include the used boundary value.) – CBroe Jul 12 '23 at 07:17

2 Answers2

2

if your php code has it as a string, not a file, then you first have to put it in a file (because PHP's libcurl api does not (yet?) support uploading files in multipart/form-data from strings), to get around this limitation you can do something like

function stringToFile(string $data): array
{
    $ret = array(
        "handle" => tmpfile(),
        "path" => null,
    );
    $ret['path'] = stream_get_meta_data($ret['handle'])['uri'];
    fwrite($ret['handle'], $data);
    rewind($ret['handle']);
    return $ret;
}

then change your code to

$url = 'https://example.com/rest';
$xml = '<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<request xmlns="http://example.com"><search path="report_date" value="2023-07-01T00:00:00+0000"/>'; //and so on..
$xmlAsFile = stringToFile($xml);
$ch = curl_init();

curl_setopt($ch, CURLOPT_POST, true);
curl_setopt($ch, CURLOPT_POSTFIELDS, array(
    'xmlRequest' => new CURLFile($xmlAsFile['path'], 'application/xml', 'file.xml')
));
(...)

also you need to remove this:

curl_setopt($ch, CURLOPT_HTTPHEADER, array('Content-Type: text/xml'));

in multipart/form-data-requests, the Content-Type is supposed to be something like Content-Type: multipart/form-data; boundary=------------------------e74d6203325745a2

but libcurl generate this header for you automatically, and you risk corrupting the POST request, making it invalid and un-parsable, if you change the outer Content-Type yourself.

Fix those 2 things and your code should work.

Meanwhile your file.xml will have it's own "inner" Content-Type, here is a full sample with both the outer and inner Content-Type:

POST / HTTP/1.1
Host: localhost:9999
User-Agent: curl/8.1.2
Accept: */*
Content-Length: 203
Content-Type: multipart/form-data; boundary=------------------------0e405b89d0ec7f23

--------------------------0e405b89d0ec7f23
Content-Disposition: form-data; name="file"; filename="test.xml"
Content-Type: application/xml

<xml></xml>

--------------------------0e405b89d0ec7f23--
  • as you can see, there is both an outer, and an inner, Content-Type, and the one you were overwriting with your curl_setopt($ch, CURLOPT_HTTPHEADER-call was the outer Content-Type (the wrong one)

(Some variation of this question has been asked many times before, but I can't find a duplicate right now, strange)

hanshenrik
  • 19,904
  • 4
  • 43
  • 89
  • I also placed an answer and basically came to the same conclusions. Also the tempfile() based solution you have in your question looks legit to me. As you asked _"yet?"_, since PHP 8.1 there is: **CURLStringFile**. And I also had problems to find a matching duplicate as this question basically asks how to convert the curl command line to PHP curl which can have many answers. Regarding how to file upload a string, I found this Q&A insightful: https://stackoverflow.com/a/72323650/367456 – hakre Jul 12 '23 at 09:22
  • This works. Thank you :) – Pekzz Jul 12 '23 at 10:07
1

What you can say as a rule of thumb is that you look for each of the curl(1) command line options (+ arguments) and operands and find the appropriate curl_setopt($option, $value).

So you can compare the command-line against the PHP code to locate potential issues that may cause the 500 error (ref). Otherwise, you can't find out what is wrong in the PHP code while the command-line works.

In the following answer, I'll show how this is done step-by-step. And it has PHP code examples of how to upload a file both from the file-system and from a string with the file-contents.

So let's take a look at what we've got here in that command-line:

curl.exe https://example.com/rest -u "xx:yy" -F xmlRequest=@file.xml > result.csv
  1. https://example.com/rest (URL)

    The URL syntax is protocol-dependent. You find a detailed description in RFC 3986. (ref)

  2. -u "xx:yy" (<user:password>)

    Specify the username and password to use for server authentication. (ref)

  3. -F xmlRequest=@file.xml (<name=content>)

    For HTTP protocol family, this lets curl emulate a filled-in form in which a user has pressed the submit button. This causes curl to POST data using the Content-Type multi‐part/form-data according to RFC 2388.

    This enables uploading of binary files etc. To force the 'content' part to be a file, prefix the file name with an @ sign. To just get the content part from a file, prefix the file name with the symbol <. The difference between @ and < is then that @ makes a file get attached in the post as a file upload, while the < makes a text field and just get the contents for that text field from a file (ref)

Then let's see which curl_setopt() operations there are in the PHP code:

$ch = curl_init();
curl_setopt($ch, CURLOPT_USERPWD, 'xx:yy');
curl_setopt($ch, CURLOPT_URL,$url);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
curl_setopt($ch, CURLOPT_TIMEOUT, 10);
curl_setopt($ch, CURLOPT_POST, true);
curl_setopt($ch, CURLOPT_POSTFIELDS, "xmlRequest=$xml");
curl_setopt($ch, CURLOPT_HTTPHEADER, array('Content-Type: text/xml'));
curl_setopt($ch, CURLOPT_FOLLOWLOCATION, true);
curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, false);
curl_setopt($ch, CURLOPT_SSL_VERIFYHOST, false);
  1. curl_setopt($ch, CURLOPT_USERPWD, 'xx:yy'); - OK, matches 2.) above.
  2. curl_setopt($ch, CURLOPT_URL,$url); - OK, matches 1.) above.
  3. curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1); - OK, your config. (As you've written, you want to do this later.)
  4. curl_setopt($ch, CURLOPT_TIMEOUT, 10); - OK, your config. (Also good for faster testing.)
  5. curl_setopt($ch, CURLOPT_POST, true); - OK, mirrors the POST method matching 3.) above.
  6. curl_setopt($ch, CURLOPT_POSTFIELDS, "xmlRequest=$xml"); - NOT OK, does not match 3.) above. 3.) above does a file-upload, here you're setting a form string value. This is never a file-upload. Interesting, let's look into that.

I can imagine this could trigger a 500 on the remote side, as they could have little error handling and the process just crashes, hence giving an 500 Internal Server Error and not providing information of what in detail is wrong with the request itself (400-499 HTTP status code range).

Let's see how to turn this into a file-upload. There are two variants: From string or from pathname.

PHP CURL File Upload from String

You perhaps want to create a file-upload from the string so that the data of the file-upload is the string contents.

If that is the case, this is how it works in PHP Curl tailored to your -F xmlRequest=@file.xml curl(1) option and argument example:

curl_setopt($ch, CURLOPT_POSTFIELDS, [
    'xmlRequest' => new CURLStringFile($xml, 'file.xml'),
    #     ^                              ^       ^
    #     |                              |       `-- upload filename
    # name of the form-field             |
    #                                    |
    #                   string contents of the file (plain binary data)
]);

Cf. CURLStringFile; answer to Send string as a file using curl and php and curl_setopt(CURLOPT_POSTFIELDS) (it already mentions CURLStringFile); CURLStringFile is available since PHP 8.1.

PHP CURL File Upload from Pathname

If that is not the case, and you want to upload the file from the file-system (instead of the string data), this is how it works in PHP Curl tailored to your -F xmlRequest=@file.xml curl(1) option and argument example:

curl_setopt($ch, CURLOPT_POSTFIELDS, [
    'xmlRequest' => new CURLFile('file.xml'),
    #     ^                          ^
    #     |                          `-- upload filename
    # name of the form-field  
]);

Cf. CURLFile; CURLFile is available since PHP 5.5.

Let's continue with the next lines:

  1. curl_setopt($ch, CURLOPT_HTTPHEADER, array('Content-Type: text/xml')); - NOT OK, the original curl command line does not have it. Remove it.

  2. curl_setopt($ch, CURLOPT_FOLLOWLOCATION, true); - NOT OK, the original curl command line does not have it. Remove it.

  3. curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, false); - NOT OK, the original curl command line does not have it. Remove it.

  4. curl_setopt($ch, CURLOPT_SSL_VERIFYHOST, false); - NOT OK, the original curl command line does not have it. Remove it.

As you have already written, you're not yet interested in converting the redirect to file (the > result.csv at the end of the command-line), so this should already be it for the 500 error you're facing.

Discussion

The technique to compare the Curl command-line with the PHP Curl code is a good procedure as it allows a straight forward way for troubleshooting and can always be completed.

It mustn't mean that it will solve all problems; however, if completed and an error persists, it can be more easily localized and configuration differences between curl on the command-line and the PHP Curl extension identified. For such problems, it is often necessary to review all the curl_setopt() calls (and their order), which is already part of the procedure and therefore enables you to handle higher level problems as well.

The answer given may also show how to port (translate) a Curl command-line to PHP Curl, however, it does not show how to find the right PHP Curl options (curl_setopt() has the listing of all the options) and their appropriate order (setting some options influence or reset others) in detail. You can also find quite some Q&A on site about the topic in general, both for Curl on the command-line and in PHP Curl.

For example, PHP - Debugging Curl shows how to obtain verbose and connection information for PHP Curl.

Also, tools exist to genereate boilerplate PHP Code from a curl command-line, for example Curl-to-PHP, but watch out, such tooling can be limited (this one is from 2017, in PHP that date is the year of the PHP 7.2 release on 30th of November and with that the PHP Curl extension required libcurl version 7.10.5 which is quite dated).

Extra: If you've got Composer in your tool-belt, you can run composer diagnose which shows the PHP version you've got at hand on the command-line as well as the Curl version of the PHP Curl extension (ext-curl is its name of the platform package in Composer):

> composer diagnose
...
PHP version: 8.2.8
...
cURL version: 7.81.0 libz 1.2.11 ssl OpenSSL/3.0.2
...

(This is similar to php_version() and curl_version() which you can make use of with PHP itself.)

hakre
  • 193,403
  • 52
  • 435
  • 836
  • Hi, I changed `$xml = 'file.xml';` and `curl_setopt($ch, CURLOPT_POSTFIELDS, [ 'xmlRequest' => new CURLStringFile($xml, 'file.xml'),]);` I think it went through but did not read the content of file.xml correctly. So it give me wrong result: string(29) "REPORT_FAILED null ". – Pekzz Jul 12 '23 at 10:14
  • Hi again, now I'm struggling with this `> result.csv`. The data is a very big string. And when I convert into array, just become 1 big value with everything in it. Can you help please? – Pekzz Jul 12 '23 at 13:30
  • @Pekzz: For your first comment: Thanks for bringing this up, this is **CURLFile** the friend of **CURLStringFile** in the answer. As both are close friends, and they shouldn't be separated from each other, I edited the answer to show this alternative. The second comment is not so clear to me, why not store it as a file first, then operate on that file? This helps you to keep problems apart, and makes it easier to obtain help as you're just concerned about parsing and processing it in PHP, and it also depends on the format (CSV you have, right?) – hakre Jul 13 '23 at 08:07
  • 1
    Wow..!! `CURLFile('file.xml')` this works very well and much shorter codes. Thank you very much for coming back and explaining all of that :) I'm not a programmer, just an enthusiast. I scrapped codes from mostly this website to learn and create simple applications for fun or to make my job easier. Thanks to people like you who care to explain and troubleshoots, I can learn so much.. For my 2nd question, I had it solved [link]https://stackoverflow.com/questions/76672715/php-create-csv-file-from-post-data-curl-execch-that-contains-only-1-big/76676195#76676195 Thank you. – Pekzz Jul 13 '23 at 11:03