0

My curl command used to give a html response from this public site till a couple of days back link here

However now I dont get any html response, even though it works from the browser. They say that they moved to the cloudflare. How can I know the server and make my curl command work? Here is my current curl output(It is a public site so any one can try)

curl -v http://f7.masaladesi.com/login.php                                                                                                                                      ~
* STATE: INIT => CONNECT handle 0x800481c0; line 1034 (connection #-5000)
* Added connection 0. The cache now contains 1 members
*   Trying 104.25.171.117...
* STATE: CONNECT => WAITCONNECT handle 0x800481c0; line 1087 (connection #0)
* Connected to f7.masaladesi.com (104.25.171.117) port 80 (#0)
* STATE: WAITCONNECT => DO handle 0x800481c0; line 1229 (connection #0)
> GET /login.php HTTP/1.1
> User-Agent: Mozilla/5.0 (Windows; U; MSIE 9.0; WIndows NT 9.0; en-US))
> Host: f7.masaladesi.com
> Accept: */*
>
* STATE: DO => DO_DONE handle 0x800481c0; line 1314 (connection #0)
* STATE: DO_DONE => WAITPERFORM handle 0x800481c0; line 1441 (connection #0)
* STATE: WAITPERFORM => PERFORM handle 0x800481c0; line 1454 (connection #0)
* HTTP 1.1 or later with persistent connection, pipelining supported
< HTTP/1.1 301 Moved Permanently
< Date: Wed, 02 Aug 2017 04:57:21 GMT
< Transfer-Encoding: chunked
< Connection: keep-alive
< Cache-Control: max-age=3600
< Expires: Wed, 02 Aug 2017 05:57:21 GMT
< Location: https://f7.masaladesi.com/login.php
* Server cloudflare-nginx is not blacklisted
< Server: cloudflare-nginx
< CF-RAY: 387e74b4538488b4-BOM
<
* STATE: PERFORM => DONE handle 0x800481c0; line 1626 (connection #0)
* Connection #0 to host f7.masaladesi.com left intact
* Expire cleared
Suresh
  • 1,081
  • 4
  • 21
  • 44

1 Answers1

1

CloudFlare uses various methods to try to determine if the end user is a "real" browser and not malicious; this is largely in an effort to mitigate DDoS attacks.

Long story short, it is not possible to reliably script access to a CloudFlare protected site. There are various temporary workarounds involving setting curl's user agent and using a "real" browser to access the site from a given IP before using curl, but they will tend to fail over time.

For a CloudFlare employee weighing in on this, look no further than here.

madscientist159
  • 550
  • 4
  • 12