0

okey I made a script checking if file exist but the header I got from Curl is bad

  $curl = curl_init( $url );
  curl_setopt( $curl, CURLOPT_NOBODY, true );
  curl_setopt( $curl, CURLOPT_HEADER, true );
  curl_setopt( $curl, CURLOPT_RETURNTRANSFER, true );
  curl_setopt( $curl, CURLOPT_FOLLOWLOCATION, true );
  curl_setopt( $curl, CURLOPT_FRESH_CONNECT, true);
  $data = curl_exec( $curl );
  var_dump($data);

and it gets me response 200 while @file_headers() gets me boolean false, which is right file cant be downloaded (connection reset by peer)

i'd use file_headers but its muuuch slower than curl (checking about 10k files)

any idea how to fix curl to force it showing it correctly if file exist?

FYI curl_getinfo($curl, CURLINFO_HTTP_CODE); shows 200 too

*I mean if file is reset by peer, because cURL detects if file exist flawlessly (making Head request) while get_headers() use get request by default, is there a way to force cURL make get request?

after adding curl_setopt( $curl, CURLOPT_HTTPGET,true); When i log curl I see: * Empty reply from server, while getting "reset by peer" file but this is downloading my whole file

ok I think Ive made workaround adding curl_setopt( $curl, CURLOPT_RANGE, "1-1"); and downloading 1 byte instead getting header and if false then file does not exist or reset by peer

klystianek
  • 21
  • 4

1 Answers1

0

The reset happens after de http status, so yes, 200 status is possible.

The error seems a forbid or slow handling downloads.

Do this:

  • Specify an user agent in your curl request
  • Set a (random) waiting time between the requests

See the following for examples: PHP cURL how to add the User Agent value OR overcome the Servers blocking cURL requests?

Community
  • 1
  • 1
schellingerht
  • 5,726
  • 2
  • 28
  • 56
  • no I just simplified the code here, I got useragent added, and the reset by peer is right answer for this file, I just want to catch this kind of error by cURL, im testing one file only Using curl I get: HTTP/1.1 200 OK Server: nginx Date: Thu, 06 Aug 2015 09:25:05 GMT Content-Type: application/octet-stream Content-Length: 2147483648 Last-Modified: Sun, 10 May 2015 23:05:19 GMT Connection: close and I sould catch error instead of 200 – klystianek Aug 06 '15 at 09:23
  • Like I said, connection reset has no effect for http status. Secondly, a http status 200 is no guarantee that the file exists ;-) – schellingerht Aug 06 '15 at 09:26
  • so how to check it properly by curl if file exist (even in cases where connection is reset by peer) ?? file_headers() answer is good for me I can catch this error with ease, but it is slow as hell, cURLing is about 20-30% faster for me. now I realized that my main question should be different :) – klystianek Aug 06 '15 at 09:28
  • wait again my words are wrong, cURL actually detects properly if file exist but does not detect if connection is reset by peer, how to detect this error? – klystianek Aug 06 '15 at 09:34
  • Check this: http://stackoverflow.com/questions/981954/how-can-one-check-to-see-if-a-remote-file-exists-using-php – schellingerht Aug 06 '15 at 09:45
  • I did earlier and half of this is based on file_headers which works good but slighty slow, and the other half on curl which only checks if file exist (does not check if connection is reset by peer while downloading this file). the only way to fix this I get chunk (10kb) of file by curl and catch error but it would be no faster I think this way (CURLOPT_RANGE and CURLOPT_WRITEFUNCTION) – klystianek Aug 06 '15 at 10:13