25

I have a site which I want to load using cURL. However, the site I'm talking about is using cloudflare protection. While on cURL request => it says 'checking your browser, you will be redirected in 5 seconds' and I'm getting redirected to the page that doesn't exists because as I think I didn't pass some protection test.

I've been looking for my answer for a long and i found only two useful links:

Link 1

Link 2

But still I can't figure it out. Any help for me?

Community
  • 1
  • 1
Marcin
  • 994
  • 2
  • 11
  • 26
  • 2
    You shouldn't be reading the url programatically then. – developerwjk Aug 28 '13 at 23:48
  • 2
    perhaps if they don't want you to access there site site way, you shouldn't? –  Aug 28 '13 at 23:51
  • Possible duplicate of [curl: can't fetch rss from website because of CloudFlare](http://stackoverflow.com/questions/11886711/curl-cant-fetch-rss-from-website-because-of-cloudflare) – andree Dec 07 '16 at 13:10

4 Answers4

30

That would be CloudFlare's I'm Under Attack Mode --> http://blog.cloudflare.com/introducing-im-under-attack-mode

If you are the owner of the site you should whitelist your own IP address. If you're a visitor to the site either

1.) The website owner has the feature enabled on their own website in which case it's intentional and there is no way around that as they are protecting their website or

2.) You are loading the site too often -- and you're triggering CloudFlare's DDOS protection.

p.s. Disclaimer: I work for CloudFlare.

Muhammad Hassaan
  • 7,296
  • 6
  • 30
  • 50
xxdesmus
  • 1,293
  • 9
  • 16
  • 4
    Hi there as you are working for CloudFlare i would like to tell you that is a way to bypass your "Content Protection".: 1. Load page with Curl, 2. Pass it on to Selenium, Set Browser to Chrome. 3. Load page with previous cookies, Wait 5 sec, and Print out its contents. EASY :) – Ziomuś Gorliczanin Jan 04 '18 at 00:56
  • @ZiomuśGorliczanin Is it still working? Do you have a code example? Thanks. – joseantgv Mar 06 '23 at 07:03
23

You can get around the in-under-attack-mode by selecting 'Copy as cUrl' in the network section under developer tools of any major browser.

This copies all the required cookies so your curl can be 'authenticated'. How long these cookies last i would assume would be up to cloudflare.

SideNote: The curl must be run on the same IP as you were loading the site with. Also for any real test/developing environment I would suggest selenium and/or phantomjs which are alot better and easily scriptable languages for website testing.

curl 'https://ilikeurls.net/ourpage.php?do=command' \
    -H 'Host: ilikeurls.net' \
    -H 'User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:33.0) Gecko/20100101 Firefox/33.0' \
    -H 'Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8' \
    -H 'Accept-Language: en-US,en;q=0.5' \
    -H 'Referer: https://ilikeurls.net/outpage.php' \                                                    
    -H 'Cookie: all required cookies will appear here' \
    -H 'Connection: keep-alive'\
    --compressed

Though you might be better off using selenium or phantomjs to script out these tests.

ticoombs
  • 333
  • 2
  • 4
  • 10
    actually, don't do `-H 'Accept-Encoding: gzip, deflate'` - do `--compressed` - this will make curl send the same header, with all compression schemes that curl was compiled with - which usually boils down to `gzip,deflate` - and curl will automatically decompress it for you too, keeping the readability & greppability, and keeping the speed improvements provided by the compression :) – hanshenrik Jul 28 '17 at 11:22
  • 11
    This trick doesn't work as of today, cloudflare returns the captcha page. – pts Jun 24 '21 at 14:43
  • 1
    @pts Can confirm, it is not working anymore, even with cookies. – miran80 Aug 10 '21 at 17:41
  • @miran80 Did you find a workaround? – terry Feb 21 '22 at 12:23
  • This answer helped in my case. I guess it worked because of `cf_` cookies. – Mike Shiyan Mar 01 '22 at 20:30
0

You might try adjusting your bot settings or finding a way to whitelist CURL.

All my curl requests started returning 403 errors after I adjusted Cloudflare settings at Firewall > Bot. All were permitted; I set Automated to Block and Likely Automated to Challenge and it broke CURL.

Slam
  • 3,125
  • 1
  • 15
  • 24
0

I don't know how to do it in php but https://github.com/lwthiker/curl-impersonate/tree/v0.5.4 would help. Maybe you can load the shared binary lib to php or use the command line tool.

xinyu
  • 118
  • 6