PHP Cannot get the text from an URL (file_get_content, CURL did not work)

Question

I am trying to get the text from the following url. http://stp.stheadline.com/data/indexNewsMarquee.json

I have tried several methods but none of them worked. I am really desperate right now please help me and thanks in advance.

P.S. I have enabled allow_url_fopen in my ini.php already. P.S. I am using XAMPP v3.3.2 and PHP v5.6.23

the following are the codes that I have tried (and failed)

CURL

$ch = curl_init();
curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, false);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
curl_setopt($ch, CURLOPT_URL, $url);
$result = curl_exec($ch);
curl_close($ch);
echo $result;

result: Empty string

file_get_contents

echo file_get_contents($url, true);

result: failed to open stream: HTTP request failed!

readfile

echo readfile($url)

result: output the address itself

See the following link. No need of curl function. http://stackoverflow.com/questions/19758954/get-data-from-json-file-with-php — Nithyanandhan M, Aug 11 '16 at 09:02

score 2 · Answer 1 · answered Aug 11 '16 at 08:58

2

They are requiring a useragent to be present in the request, so try:

$url='http://stp.stheadline.com/data/indexNewsMarquee.json';
$ch = curl_init();
curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, false);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
curl_setopt($ch, CURLOPT_USERAGENT, 'banana');
curl_setopt($ch, CURLOPT_URL, $url);
$result = curl_exec($ch);
curl_close($ch);
echo $result;

answered Aug 11 '16 at 08:58

Professor Abronsius

33,063
5
32
46

I'm genuinely curios how'd you figure out the need useragent in the request...? – Andrei Aug 11 '16 at 09:00
Experience of trying curl requests has shown that on many occasions when a request has failed silently that adding a useragent makes it work ( as it did in this request ) ~ presumably because some bots do not have a useragent and thus do not appear as valid browser traffic. – Professor Abronsius Aug 11 '16 at 09:02
Huh, well. Good to know. – Andrei Aug 11 '16 at 09:02

score 2 · Accepted Answer · edited Aug 11 '16 at 09:14

2

Hi you have to pass header in your curl request, so server can treat its as browser request.

 $url="http://stp.stheadline.com/data/indexNewsMarquee.json";

 $requestHeaders = array(
    "Accept:application/json, text/javascript, */*; q=0.01",
    "Accept-Language:en-US,en;q=0.8",
    "Connection:keep-alive",
    "Host:stp.stheadline.com",
    "Origin:http://stp.stheadline.com",
    "Referer:http://stp.stheadline.com/data/indexNewsMarquee.json",
    "User-Agent:Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/49.0.2623.87 Safari/537.36"
 );

 $ch = curl_init();
 curl_setopt($ch, CURLOPT_URL, $url);
 curl_setopt($ch, CURLOPT_HTTPHEADER, $requestHeaders);
 curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
 $output = curl_exec($ch);
 curl_close($ch);
 echo $output;

edited Aug 11 '16 at 09:14

Riad

3,822
5
28
39

answered Aug 11 '16 at 09:07

Haresh Vidja

8,340
3
25
42

Thanks it works. can you explain a bit what I did wrong (what does it mean to pass the header) I am quite new to this so I would really appreciate it if you can elaborate a little.Thanks in advance – yploo Aug 11 '16 at 09:17
Yeah @Kevin, it seems http://stp.stheadline.com has implemented security for prevent access unkown requests on server. By including headers in curl request like origin, referer, urger agent.. so that server ensures that request come from browser... it like some tricks to use CURL request as browser request :).. I hope you like my answer and give up vote and mark as correct... thanks in advance – Haresh Vidja Aug 11 '16 at 09:21
You can refere more from https://www.sitepoint.com/using-curl-for-remote-requests/ – Haresh Vidja Aug 11 '16 at 09:24
thanks for your explanation. How can determine if the website has the security implementation. Just curious. Thank you. – yploo Aug 12 '16 at 02:19

score 0 · Answer 3 · answered Aug 11 '16 at 09:01

0

Use Just like this:

 $curl_handle=curl_init();
 curl_setopt($curl_handle,CURLOPT_URL,'Your URL');
 curl_setopt($curl_handle, CURLOPT_CONNECTTIMEOUT, 2);
 curl_setopt($curl_handle, CURLOPT_RETURNTRANSFER, 1);
 curl_setopt($curl_handle, CURLOPT_USERAGENT, 'Your application name');
$query = curl_exec($curl_handle);
curl_close($curl_handle);
print_r($query);

I checked this Code from Your URL.

answered Aug 11 '16 at 09:01

Dharmendra

127
12

and Leave Section of "Your Application Name" and tried out – Dharmendra Aug 11 '16 at 09:02
Thanks it works. In your code, there are (**CURLOPT_USERAGENT** and **CONNECTION_TIMEOUT**(which I dont have). Can you tell me what the code did? Thanks in advance. – yploo Aug 11 '16 at 09:20
@Kevin 1. CONNECTTIMEOUT : The number of seconds to wait while trying to connect. (0 For infinite). 2. USERAGENT : The contents of the "User-Agent: " header to be used in a HTTP request. – Dharmendra Aug 19 '16 at 11:09

Rohit Dhiman · Answer 4 · 2016-08-11T09:43:59.990

Try this out.

$ch = curl_init();
curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, false);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
curl_setopt($ch, CURLOPT_URL, 'http://stp.stheadline.com/data/indexNewsMarquee.json');
$result = curl_exec($ch);
curl_close($ch);

echo($result); // Json data

/*$obj = json_decode($result); // To get result in array
print_r($obj);*/

Or

$url = 'http://stp.stheadline.com/data/indexNewsMarquee.json';
$ch = curl_init();
curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, false);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
curl_setopt($ch, CURLOPT_URL, $url);
$result = curl_exec($ch);
curl_close($ch);

echo($result); // Json data

It's working for me :)

score 0 · Answer 5 · answered Aug 11 '16 at 09:26

Kevin,

Please try below Code.

header('Content-Type: text/html; charset=utf-8');

$url='http://stp.stheadline.com/data/indexNewsMarquee.json';

$ch = curl_init();
curl_setopt($ch, CURLOPT_HTTPHEADER, array('text/html; charset=utf-8'));
curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, false);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
curl_setopt($ch, CURLOPT_ENCODING, "");
curl_setopt($ch, CURLOPT_URL, $url);
$result = curl_exec($ch);
curl_close($ch);
$final_result = json_decode($result);

echo "<pre>";
print_r($final_result);

I have added header('Content-Type: text/html; charset=utf-8'); and curl_setopt($ch, CURLOPT_ENCODING, ""); in your code, and it will work well, also will display correct characters as well.

PHP Cannot get the text from an URL (file_get_content, CURL did not work)

CURL

file_get_contents

readfile

5 Answers5