Why does CURL's PUT fail authentication before uploading the payload, but XHR PUT only after?

Question

I'm working with a rest API with token based authentication where some users have permissions to upload a file and some don't.

The problem is when some user without permission to upload a file tries to upload (Say a 1GB file), I'm getting the error response only after the whole of 1GB is uploaded.

If I copy the request as curl from chrome developers tools and send it via terminal it fails immediately.

I tested the curl command with the token of user who has permissions to upload, it works as expected.

So, How is curl different from XHR?

Curl is synchronous, and XHR is not by default. I tried making XHR synchronous, but it still has to upload the whole file before it got a response.

function upload(file, url) {
    var xhr = new XMLHttpRequest();
    xhr.upload.key = xhr.key = file.key
    xhr.upload.addEventListener("progress", updateProgress);
    xhr.addEventListener("error", transferFailed);
    xhr.addEventListener("abort", transferCanceled);
    xhr.open("PUT", url);
    xhr.setRequestHeader("Content-Type", "application/octet-stream");
    xhr.setRequestHeader("X-Auth-Token", service.token.token);


    xhr.addEventListener("readystatechange", function (e) {
      if (this.readyState === 4 && this.status >= 200 && this.status < 300) {
        transferComplete(e)
      }
      if (this.readyState === 4 && this.status === 0) {
        transferFailed(e)
      }
      if (this.status >= 400) {
        transferFailed(e)
      }

    });


    xhr.send(file);
}

Here is the exact curl command, formatted for readability:

curl 'https://my-website.com/54321/my-filename.jpg' 
  -X PUT   
  -H 'Pragma: no-cache' 
  -H 'Origin: https://my-website.com' 
  -H 'Accept-Encoding: gzip, deflate, sdch, br' 
  -H 'Accept-Language: en-US,en;q=0.8' 
  -H 'User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_12_3) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/56.0.2924.87 Safari/537.36' 
  -H 'Content-Type: application/octet-stream' 
  -H 'Accept: */*' 
  -H 'Cache-Control: no-cache' 
  -H 'X-Auth-Token: fsadgsdgs' 
  -H 'Referer: https://some-allowed-origin-referrer.com/' 
  -H 'Connection: keep-alive' 
  -H 'Content-Length: 86815' 
  -F "data=@/Users/satish/abc.jpg" --compressed --insecure 

//Headers stripped except for token

  curl 'https://my-website.com/54321/my-filename.jpg' 
    -X PUT  
    -H 'X-Auth-Token: fsadsdsdaf' 
    -F "data=@/Users/satish/abc.jpg" 
    --compressed --insecure 

--- Update 21-02-2017 ---

To rule out any API end point specific behaviour, I wrote a crude PHP script to test this observation and it is still true. Below is the php script I tried uploading to.

<?php
/**
 * If I comment out the below lines, then curl is failing immediately.
 * But XHR doesn't
 **/

// http_response_code(400);
// return;

/* PUT data comes in on the stdin stream */
$putdata = fopen( "php://input", "r" );

/* Open a file for writing */
$file = fopen( "/Users/satish/Work/Development/htdocs/put-test/wow.jpg", "w" );

/* Read the data 1 KB at a time
   and write to the file */
while ( $data = fread( $putdata, 1024 ) ) {
    fwrite( $file, $data );
}

/* Close the streams */
fclose( $file );
fclose( $putdata );
?>

Answer 1

When curl has a PUT request with authentication required, it first sends a “probe” with no content to allow the server the opportunity to refuse the connection before any data gets sent.

This doesn’t seem to be documented anywhere other than a comment in the code in Curl_http() in lib/http.c.

(Note that in versions of curl prior to 7.53.0 (2017‑02‑22, i.e. newer than the question), there was a bug whereby the user-supplied Content‑Length header (or no header) was sent instead of Content‑Length: 0.)

XHR implements no such probe and simply sends all the content with the initial request.

Answer 2

Try adding -H "Expect:" to your curl command.

I might be wrong but here's my hunch:

• XHR: Expect is a forbidden header name
• curl: adds Expect: 100-continue by default