0

I am trying to use a none browser method to fetch a webpage and API. I use Chome incognito mode and request the site and view the request in the inspector and "copy as cURL".

enter image description here

I paste the command in a terminal window and run but get blocked by Cloudflare with a return code of 403.

enter image description here

enter image description here

I look in Proxyman and the request looks identical.

This is a recent change on the target site or Cloudflare side as I was able to curl to their pages and API not issue until 2 days ago.

I can, however, "copy as cURL" in Proxyman and run the cURL command and it works with the only difference being --proxy http://localhost:9090is added to the end of the request. While Proxyman is running it runs this proxy. Shouldn't this mean I should be able to run the cURL command somehow without the Proxyman's proxy?

Things I've tried.

  • Ran with cURL on the command line. (doesn't work)

  • Ran with cURL in codeRunner. (doesn't work)

  • Ran with cURL with PHP curl with identical headers. (doesn't work)

  • Re-ran cURL request in Proxyman from failed cURL request on the command line. (works)

  • Used VPN and retried all the above. (same results)

  • ssh'ed into 2 different remote servers a retried all the above cURL commands (doesn't work)

For the failed calls they are all Cloudflare blocking with 403 status.

Again, this is new behavior as I was using these methods fine before a day or two ago. I assume WAF strong request fingerprinting is being used or something. Please let me know if I am missing any crucial step or other methods I can try.

A related question I tried was here

ChadH
  • 303
  • 3
  • 13
  • Yes they probably detected a scraper and turned up their filtering level to block it. "it" being you, and CloudFlare being very good at this. I would suggest contacting the site operator and working out an arrangement to access their data. – Sammitch Apr 14 '21 at 17:29
  • 1
    Yea. I'm still wondering how I can successfully run the cURL command through Proxyman though. What is it doing differently than running it normally. – ChadH Apr 14 '21 at 17:32

0 Answers0