PHP file_get_contents convert & into &

Question

The mentioned link in variable $link is redirect to product page. Its working fine when i call/open it in browser. But it does not work in PHP file_get_contents function.

My Code:

    $url = "750651";
    $link = "http://www.costco.com/CatalogSearch?storeId=10301&catalogId=10701&langId=-1&keyword=$url";
    $link = str_replace('&amp;','&',$link);
    $res = file_get_contents(html_entity_decode(urldecode($link)));

Error

Warning: file_get_contents(http://www.costco.com/CatalogSearch?storeId=10301&amp;catalogId=10701&amp;langId=-1&amp;keyword=750651): failed to open stream: HTTP request failed! HTTP/1.0 403 Forbidden

How can I prevent conversion of & into & in file_get_contents functions I also have tried following code but no success

 $link = "http://www.costco.com/CatalogSearch?";
 $options = array("storeId"=>"10301","catalogId"=>"10701","langId"=>"-1","keyword"=>$url);
 $link .= http_build_query($options,'','&');
 $res = file_get_contents($link);

The link you have responds with a 302, it redirect to another link. This is the reason you are having problems. `http://www.costco.com/Perfect-Smile%C2%AE-EASY-CLICK-Teeth-Whitening-Pen.product.100036334.html?catalogId=10701&keyword=750651&langId=-1&storeId=10301` — Ronni Skansing, Sep 11 '14 at 17:43
@RonniSkansing Yes it is redirecting so need next page content. How is it possible? — DJ MHA, Sep 11 '14 at 17:46
Yes you could either look at the response and make a new request or try out the comment I added to the other answer with `curl_setopt($ch, CURLOPT_FOLLOWLOCATION, true)` also if you search "file_get_contents redirect" you will find some stuff that is readable with other people asking something very similar — Ronni Skansing, Sep 11 '14 at 17:47

score 1 · Accepted Answer · edited May 23 '17 at 12:30

I also found alternative function to do this. I hope this would be use full.

function get_fcontent( $url,  $javascript_loop = 0, $timeout = 5 ) {
    $url = str_replace( "&amp;", "&", urldecode(trim($url)) );

    $cookie = tempnam ("/tmp", "CURLCOOKIE");
    $ch = curl_init();
    curl_setopt( $ch, CURLOPT_USERAGENT, "Mozilla/5.0 (Windows; U; Windows NT 5.1; rv:1.7.3) Gecko/20041001 Firefox/0.10.1" );
    curl_setopt( $ch, CURLOPT_URL, $url );
    curl_setopt( $ch, CURLOPT_COOKIEJAR, $cookie );
    curl_setopt( $ch, CURLOPT_FOLLOWLOCATION, true );
    curl_setopt( $ch, CURLOPT_ENCODING, "" );
    curl_setopt( $ch, CURLOPT_RETURNTRANSFER, true );
    curl_setopt( $ch, CURLOPT_AUTOREFERER, true );
    curl_setopt( $ch, CURLOPT_SSL_VERIFYPEER, false );    # required for https urls
    curl_setopt( $ch, CURLOPT_CONNECTTIMEOUT, $timeout );
    curl_setopt( $ch, CURLOPT_TIMEOUT, $timeout );
    curl_setopt( $ch, CURLOPT_MAXREDIRS, 10 );
    $content = curl_exec( $ch );
    $response = curl_getinfo( $ch );
    curl_close ( $ch );

    if ($response['http_code'] == 301 || $response['http_code'] == 302) {
        ini_set("user_agent", "Mozilla/5.0 (Windows; U; Windows NT 5.1; rv:1.7.3) Gecko/20041001 Firefox/0.10.1");

        if ( $headers = get_headers($response['url']) ) {
            foreach( $headers as $value ) {
                if ( substr( strtolower($value), 0, 9 ) == "location:" )
                    return get_url( trim( substr( $value, 9, strlen($value) ) ) );
            }
        }
    }

    if (    ( preg_match("/>[[:space:]]+window\.location\.replace\('(.*)'\)/i", $content, $value) || preg_match("/>[[:space:]]+window\.location\=\"(.*)\"/i", $content, $value) ) && $javascript_loop < 5) {
        return get_url( $value[1], $javascript_loop+1 );
    } else {
        return array( $content, $response );
    }
}

To see results

$lurl=get_fcontent($link);
echo $lurl[0];

Source https://stackoverflow.com/a/5402193/3466544

score 1 · Answer 2 · edited Nov 25 '15 at 06:38

1

I used it this way:

$myURL = 'http://www.costco.com/CatalogSearch?';   
$options = array("storedId"=>$10301,"câtlogId"=>10701,"langId"=>-1,"keyword"=>$url);
$myURL .= http_build_query($options,'','&');

$myData = file_get_contents("$myURL");

And it ran well. Try with this.

edited Nov 25 '15 at 06:38

Manjunath Ballur

6,287
3
37
48

answered Nov 25 '15 at 05:40

truthblue82

129
1
6

score 0 · Answer 3 · edited Sep 11 '14 at 18:37

0

Try it without the urldecode and entity_decode, or do those before the string_replacement

$link = "http://www.costco.com/CatalogSearch?storeId=10301&catalogId=10701&langId=-1&keyword=$url";
$link = str_ireplace('&amp;','&', html_entity_decode(urldecode($link)));
$res = file_get_contents($link);

You cannot get every site via file_get_contents, this because of the same-origin policy. The site owner should open up Access-Control-Allow-Origin for that to work. You can however download the site via CURL, that goes something like this:

$ch = curl_init();
$timeout = 5;
curl_setopt($ch, CURLOPT_URL, $link);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
curl_setopt($ch, CURLOPT_CONNECTTIMEOUT, $timeout);
curl_setopt($ch, CURLOPT_FOLLOWLOCATION, true); // if you want to follow redirects
$data = curl_exec($ch);
curl_close($ch);

edited Sep 11 '14 at 18:37

anon

4,578
3
35
54

answered Sep 11 '14 at 17:15

Erik Terwan

2,710
19
28

I do var_dump($data); but it returns empty string '' (length=0) – DJ MHA Sep 11 '14 at 17:20
1

How does the URL look if you echo it? You should look in the page source, not the output or inspect element feature in webkit. – Erik Terwan Sep 11 '14 at 17:21
There is nothing in page source too :( – DJ MHA Sep 11 '14 at 17:30
Do you have cURL installed? You can check by creating a new php file with the line `` – Erik Terwan Sep 11 '14 at 17:31
Yes, its installed and enabled. cURL support enabled – DJ MHA Sep 11 '14 at 17:35
`You cannot get every site via file_get_contents, this because of the same-origin policy. The site owner should open up Access-Control-Allow-Origin for that to work`, it has nothing to do same-origin policy. – Ronni Skansing Sep 11 '14 at 17:37
Add `curl_setopt($ch, CURLOPT_FOLLOWLOCATION, true)` to make it work. – Ronni Skansing Sep 11 '14 at 17:46

PHP file_get_contents convert & into &

3 Answers3