can't connect to specific website using shell_exec and curl php

Question

I am trying to access this (https://www.gearbest.com/cell-phones/pp_009969695587.html) page but with no luck. I get no response...

<?php

$output = shell_exec('curl -L "Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:59.0) Gecko/20100101 Firefox/59.0" https://www.gearbest.com/cell-phones/pp_009969695587.html');
echo $output;

  $html= str_get_html($output);
  echo $html;
  ?>

So what does your http server's error log file reveal what the issue is? — arkascha, Aug 23 '19 at 16:36
For me it works (I get an html page which says that this is not allowed) so it must be some problem with your network config. — Paflow, Aug 23 '19 at 16:43
Maybe they don't like people scraping their content? Read the site terms and conditions? But I am also not convinced that is a valid cURL command. `curl: (6) Couldn't resolve host 'Mozilla'` — ficuscr, Aug 23 '19 at 16:43
Not allowed? But its a web page what do you mean not allowed? — stefanosn, Aug 23 '19 at 16:43
No its ok to scrap i asked them. Other pages work ok. Some pages like this one dont work. — stefanosn, Aug 23 '19 at 16:44
You _are_ aware that "a web page" is a very loose term? And that it is very well possible that the operator takes actions to block certain requests to his liking? — arkascha, Aug 23 '19 at 16:48
Yeah i understand so i have to make it look like i am a browser — stefanosn, Aug 23 '19 at 16:48
Isn't `-L` location? Don't you want `-A`, as in A for agent? Still think they are blocking - why spoof the agent otherwise? — ficuscr, Aug 23 '19 at 16:50
Seriously, try stuff like this on the command line before wrapping it in a shell call. — ficuscr, Aug 23 '19 at 17:06

score 2 · Answer 1 · answered Aug 23 '19 at 17:00

2

you forgot to check stderr. you get output in stderr, but i guess your stderr is redirected someplace you're not looking (for example if you're using php-fpm+nginx, then stderr is often by default redirected to nginx's error log)

either check your stderr, or redirect stderr to stdout by adding 2>&1 at the end of the command.

for the record, here is what i expect you are getting in stderr (it's what i get in stderr when running it):

curl: (6) Could not resolve host: Mozilla
curl: (56) OpenSSL SSL_read: SSL_ERROR_SYSCALL, errno 104

.. this won't solve what i suspect is your actual problem, that gearbeast is blocking curl, but that's not really your question, so i guess i won't go into that..?

answered Aug 23 '19 at 17:00

hanshenrik

19,904
4
43
89

So its an ssl issue? – stefanosn Aug 23 '19 at 17:06
1

@stefanosn no, it's not. it appears that they're running a firewall that blocks anything that doesn't look enough like a web-browser, your curl request included. once you send all the headers, and their firewall determines that your headers does not look like a web browser's headers, it drops the connection, it's not an ssl issue. – hanshenrik Aug 23 '19 at 17:11
hmm so i have to send all headers then hans ok i will try – stefanosn Aug 23 '19 at 17:16

score 0 · Answer 2 · answered Aug 23 '19 at 17:24

Based on this answer https://stackoverflow.com/a/31597823/80836 the following seems to work:

curl 'https://www.gearbest.com/cell-phones/pp_009969695587.html' -H 'authority: www.gearbest.com' -H 'upgrade-insecure-requests: 1' -H 'user-agent: Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/76.0.3809.100 Safari/537.36' -H 'sec-fetch-mode: navigate' -H 'sec-fetch-user: ?1' -H 'accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3' -H 'sec-fetch-site: none' -H 'accept-encoding: gzip, deflate, br' -H 'accept-language: en-US,en;q=0.9,el;q=0.8' --compressed

don't set the `accept-encoding` header manually. for example, if curl was not compiled with `br` encoding support (very normal), and the server actually decides to use br encoding, then the return will be garbled unreadable binary data. when using `--compressed`, curl will set the header for you, and with all supported encodings (which is decided on curl compile time) — hanshenrik, Aug 23 '19 at 17:28

can't connect to specific website using shell_exec and curl php

2 Answers2