How to deal with Clouflare, I get Cloudflare instead of the actual site source when i do an Idhttp.get

Question

I am trying to do a simple IdHttp.get but the response i get gives me the page for CloudFlare which says Checking your browser before accessing...
How can i deal with this ?, i tried any option i could think of, i even tried doing an Sleep(6000) and repeating the IdHttp.get, since the CloudFlare message says wait for 5 second
Here is my code :

var 
  mIdHttp: TIdHttp;  
  URL: String;  
  memoryStream: TMemoryStream;
Begin  
  mIdHttp := TIdHttp.create(nil);
  mIdHttp.AllowCookies := true;
  mIdHttp.HandleRedirects := true;
  mIdHttp.Request.UserAgent := 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/57.0.2987.133 Safari/537.36 OPR/44.0.2510.1457';
  mIdHttp.Request.Accept := 'text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8';
  mIdHttp.Request.AcceptEncoding := 'gzip, deflate';
  mIdHttp.Request.AcceptLanguage := 'en-US,en;q=0.9';
  mIdHttp.Request.Host := 'somesite.com/'';  
  URL := 'https://somesite.com'';  
  //Both ssleay32.dll and libeay32.dll are beside the application.
  mIdHttp.get(URL, memoryStream);  
  memoryStream.saveToFile('response.txt');  
End;

On a side note, DO NOT set the `AcceptEncoding` manually like you are. You are telling the server that you support compression, but you really don't. Let `TIdHTTP` handle that property for you. If you want to support compression, assign a component to the `TIdHTTP.Compressor` property. — Remy Lebeau, Jul 18 '18 at 15:56

score 2 · Answer 1 · answered Sep 19 '18 at 08:28

I solved this problem，first update the indy version to 10.6.2.0,then follow my request code below:

function Request(Method,URL:String;RequestHeaders,SendString:String;TreadTLog:TLogger;ContentType:string='application/x-www-form-urlencoded'
  ;SSLVersion:TIdSSLVersion=sslvSSLv23;UserAgent:string='Mozilla/5.0 (Windows NT 6.1; WOW64; rv:53.0) Gecko/20100101 Firefox/53.0'):string;
var
  SendStream,GetSStream: TStringStream;
  IdHTTP:TIdHTTP;
  List:TStringList;
  LHandler: TIdSSLIOHandlerSocketOpenSSL;
  EvHandler:TEventHandlers;
  IdConnectionIntercept:TIdConnectionIntercept;
  i:Integer;
  RetStr,S,KEY,VALUE:string;  
begin
  Result:='';
  if URL='' then Exit;
  IdHTTP:=TIdHTTP.Create(nil);
  List := TStringList.Create;  
  SendStream:=TStringStream.Create('');
  GetSStream:=TStringStream.Create('');
  try

    ExtractStrings(['&'],[],pchar(RequestHeaders),List);
    SendStream.WriteString(SendString);
    try
      IdConnectionIntercept:= TIdConnectionIntercept.Create(nil);
      IdConnectionIntercept.OnReceive := EvHandler.IdConnectionInterceptReceive;
      IdConnectionIntercept.OnSend := EvHandler.IdConnectionInterceptSend;
      IDHTTP.Intercept := IdConnectionIntercept;   

      if (pos('HTTPS',UPPERCASE(URL))>0) then
      begin
        LHandler := TIdSSLIOHandlerSocketOpenSSL.Create(IDHTTP);
        IdHTTP.IOHandler:=LHandler;
        LHandler.OnVerifyPeer:=EvHandler.LHandlerVerifyPeer;
        LHandler.SSLOptions.Method := SSLVersion;
        LHandler.SSLOptions.SSLVersions:=[sslvSSLv23,sslvSSLv2, sslvSSLv3, sslvTLSv1,sslvTLSv1_1,sslvTLSv1_2];
        LHandler.SSLOptions.Mode := sslmUnassigned;
        LHandler.SSLOptions.VerifyMode := LHandler.SSLOptions.VerifyMode + [sslvrfPeer];;
        LHandler.SSLOptions.VerifyDepth := 0;   
      end
      else IdHTTP.IOHandler:=nil;
      for i:=0 to List.Count-1 do
      begin
        S:=Trim(List.Strings[i]);
        if S<>'' then
        begin
          KEY:=Copy(S,1,Pos('=',S)-1);
          VALUE:=Copy(S,Pos('=',S)+1,Length(S));
          IdHTTP.Request.CustomHeaders.Add(KEY+':'+VALUE);
        end;
      end;
      IdHTTP.Request.ContentType :=ContentType;
      IdHTTP.Request.UserAgent:=UserAgent;

      IdHTTP.HandleRedirects := True;
      IdHTTP.AllowCookies := True;
      IdHTTP.Request.Connection:='keep-alive';
      IdHTTP.Request.BasicAuthentication := False;
      IdHTTP.Request.Accept:='text/html, */*';
      IdHTTP.Request.AcceptEncoding:='identity';
      //IdHTTP.ReadTimeout:=MySysPM.PMA06;
      //IdHTTP.ConnectTimeout:=MySysPM.PMA06;
      IdHTTP.HTTPOptions:=IdHTTP.HTTPOptions+[hoKeepOrigProtocol];
      IdHTTP.ProtocolVersion:=pv1_1;
      IdHTTP.Request.Referer:='';  

      IF UpperCase(Method)='POST' then
      BEGIN
        IdHTTP.Post(URL,SendStream,GetSStream);
        RetStr:=Utf8ToAnsi(GetSStream.DataString);
      end
      else if UpperCase(Method)='GET' then
      begin
        RetStr:=IdHTTP.Get(URL);
      end
      else begin
        IdHTTP.Delete(URL);
      end;
      IdHTTP.Disconnect;
      Result:=RetStr
    except
      on E:exception do
      begin
        TreadTLog.WriteLog('Request:'+e.Message,1);
      end;
    end;
  finally
    try
      if Assigned(IdConnectionIntercept) then FreeAndNil(IdConnectionIntercept);
      FreeAndNil(LHandler);
      FreeAndNil(SendStream);
      FreeAndNil(GetSStream);
      FreeAndNil(List);
      FreeAndNil(IdHTTP);
    except
    end;
  end;
end;

Remy Lebeau · Answer 2 · 2018-07-18T21:00:23.593

1

Cloudflare implements protection against bots (DDoS attacks, etc), that is what the 5 second wait is about.

Redirect to a website with cloudflare 5 second protection C#

Your app is not a web browser that executes Javascript, so it gets treated as a bot instead.

Cloudflare sends a challenge in Javascript, which must be computed and sent back to Cloudflare in order to obtain a cookie that can then be used to bypass the protection on subsequent requests.

How can I get html from page with cloudflare ddos portection?

The above links are for C#. You will have to replicate a similar solution in Delphi using Indy and whatever Javascript/Regex library you want.

edited Jul 18 '18 at 21:00

answered Jul 18 '18 at 16:08

Remy Lebeau

555,201
31
458
770

Can i open the website in a `TWebBrowser` and after it passes the `CloudFlare` challenge, i pass the information to an 'IdHttp' ? if the answer is yes, how can i achieve this "transfer of cookie and other informations from TWebBrowser to `IdHttp`?" , thanks. – Ali Ahmadi Jul 18 '18 at 17:10
@AliAhmadi See [Getting Cookie from TWebBrowser](https://stackoverflow.com/questions/14826727/) – Remy Lebeau Jul 18 '18 at 21:03

How to deal with Clouflare, I get Cloudflare instead of the actual site source when i do an Idhttp.get

2 Answers2