3

I use Curl -dump-header but i get also access denied. Is there any way to connect and get html code?

Curl --dump-header - https://www.gearbest.com/car-charger/pp_009363232829.html

> HTTP/2 403  server: AkamaiGHost mime-version: 1.0 content-type:
> text/html content-length: 314 cache-control: max-age=60 expires: Sat,
> 22 Jun 2019 22:25:51 GMT date: Sat, 22 Jun 2019 22:24:51 GMT
> set-cookie: AKAM_CLIENTID=7e3530d888ae97fef4ad26c997d733c5;
> expires=Mon, 31-Dec-2038 23:59:59 GMT; path=/; domain=.gearbest.com
> vary: User-Agent

<HTML><HEAD>
<TITLE>Access Denied</TITLE>
</HEAD><BODY>
<H1>Access Denied</H1>

You don't have permission to access "http&#58;&#47;&#47;www&#46;gearbest&#46;com&#47;car&#45;charger&#47;pp&#95;009363232829&#46;html" on this server.<P>
Reference&#32;&#35;18&#46;85451502&#46;1561242291&#46;25f21039
</BODY>
</HTML>

If i go to the page using the browser it works fine. I just want to get the html code of the page. any help appreciated.

stefanosn
  • 3,264
  • 10
  • 53
  • 79
  • 1
    Did you try to change user agent to something that looks more like a browser? https://www.cyberciti.biz/faq/curl-set-user-agent-command-linux-unix/ – alx Jun 22 '19 at 22:39
  • Let me check it! – stefanosn Jun 22 '19 at 22:43
  • @alx yes this worked... curl -A "Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:59.0) Gecko/20100101 Firefox/59.0" https://www.gearbest.com/car-charger/pp_009363232829.html . Does not work in php :( $curl1 = curl_init(); $url="https://www.gearbest.com/car-charger/pp_009363232829.html"; curl_setopt($curl1, CURLOPT_URL, $url); curl_setopt($curl1, CURLOPT_CONNECTTIMEOUT, 20); curl_setopt($curl1, CURLOPT_USERAGENT, 'Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:59.0) Gecko/20100101 Firefox/59.0'); $str = curl_exec($curl1); any thoughts for php issue? – stefanosn Jun 22 '19 at 22:53
  • Can you modify your question and add your PHP code, and also any response or error message that PHP returns? – alx Jun 22 '19 at 22:56

1 Answers1

1

Here is you PHP code modified, and it works perfectly fine (tested):

<?php
$curl1 = curl_init();
$url = "https://gearbest.com/car-charger/pp_009363232829.html";
curl_setopt($curl1, CURLOPT_URL, $url);
curl_setopt($curl1, CURLOPT_CONNECTTIMEOUT, 20);
curl_setopt($curl1, CURLOPT_USERAGENT, 'Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:59.0) Gecko/20100101 Firefox/59.0');
curl_setopt($curl1, CURLOPT_RETURNTRANSFER, true);
curl_setopt($curl1, CURLOPT_FOLLOWLOCATION, true);
$str = curl_exec($curl1); 
echo $str;

Few notes:

  • you were expecting curl_exec() to return HTML code, but without setting CURLOPT_RETURNTRANSFER option first that would not happen
  • often sites use redirects (e.g. GearBest redirects to www. if you omit it), and to handle that properly you need to set CURLOPT_FOLLOWLOCATION option
alx
  • 2,314
  • 2
  • 18
  • 22
  • this is my code but check in the end the ssl error... $curl1 = curl_init(); $url="https://www.gearbest.com/car-charger/pp_009363232829.html"; curl_setopt($curl1, CURLOPT_URL, $url); curl_setopt($curl1, CURLOPT_HEADER, true); curl_setopt($curl1, CURLOPT_RETURNTRANSFER, 1); curl_setopt($curl1, CURLOPT_FOLLOWLOCATION, true); curl_setopt($curl1, CURLOPT_CONNECTTIMEOUT, 20); curl_setopt($curl1, CURLOPT_USERAGENT, 'Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.2 (KHTML, like Gecko) Chrome/22.0.1216.0 Safari/537.2'); i get error LibreSSL SSL_read: SSL_ERROR_SYSCALL, errno 60 – stefanosn Jun 22 '19 at 23:12
  • That's quite generic error, could mean broad range of things (as I can conclude from google results). Are you behind some proxy or vpn? This could be the reason. Before you ask, browsers use different SSL stack which might (or might not) react differently to network specifics. – alx Jun 22 '19 at 23:18
  • Correction: the code in my answer works as expected, I see store page HTML (previously was trying on Windows machine, it failed, tried on macOS, and it worked fine). So, I'm almost sure the problem is in your PHP/CURL/network configuration. Try different machine? – alx Jun 22 '19 at 23:21
  • I am on mac as well! this is driving me crazy. – stefanosn Jun 22 '19 at 23:24
  • You can try to install different PHP version (e.g. with `brew install php@7.3`), I think it will have bundled SSL and cURL libraries, and you might get better results. – alx Jun 22 '19 at 23:25
  • https://stackoverflow.com/questions/53127113/php-curl-headers-do-not-return-from-website i think i have the same problem – stefanosn Jun 22 '19 at 23:30
  • 1
    Well, my PHP comes from `brew`, and it works. Probably will work for you too. – alx Jun 22 '19 at 23:33
  • Well i just used $output = shell_exec('curl -A "Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:59.0) Gecko/20100101 Firefox/59.0" '.$url); and it works! so the problem is in php i guess because out of php it works. i am confused! – stefanosn Jun 23 '19 at 14:01
  • Hmm, you already linked that SO question above which seems to perfectly explain the situation. – alx Jun 23 '19 at 14:02