Getting a 404 while using PHP Curl but can visit the url

Question

I'm trying to use curl to get data from a specific website and am getting a 404 as response. The curious thing is that i can visit the url just fine.

I guess the problem lies elsewhere (likely with my malformed request to start with) and was wondering if someone could shed some light:

    ...
    curl_setopt($ch, CURLOPT_TIMEOUT, 30);
    curl_setopt($ch, CURLOPT_USERAGENT, 'my own agent');
    curl_setopt($ch, CURLOPT_SSL_VERIFYHOST, 0);
    curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, 0);
    curl_setopt($ch, CURLOPT_URL, $url);
    if (! empty($data)) {
        curl_setopt($ch, CURLOPT_POST, 1);
        curl_setopt($ch, CURLOPT_POSTFIELDS, $data);
    }
    curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
    curl_setopt($ch, CURLOPT_FOLLOWLOCATION, 1);
    curl_setopt($ch, CURLOPT_COOKIEJAR, 'c4l.cookie');
    $output['returned'] = curl_exec($ch);
    $output['info'] = curl_getinfo($ch);

    return $output;

PS: The above code was not written by me nor I'm that good with PHP but I need to need to debug and find whatever is causing the error.

Any help would be much appreciated.

EDIT:

The url is: http://www.example.com/something/else/here.also

EDIT2:

The actual response is:

<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
<HTML><HEAD>
<TITLE>404 Not Found</TITLE>
</HEAD><BODY>
<H1>Not Found</H1>
The requested URL /something/else/here.also was not found on this server.<P>
<HR>
<ADDRESS>Oracle-Application-Server-10g/10.1.3.5.0 Oracle-HTTP-Server     Server at cbl08901app01.nat.bt.com Port 61000</ADDRESS>
</BODY></HTML>

Did you try `echo`-ing the `$url` to see if it's actually valid? — Emanuel Vintilă, Feb 23 '16 at 14:43
*To debug:* Add error reporting to the top of your file(s) right after your opening PHP tag for example ` — Funk Forty Niner, Feb 23 '16 at 14:44
We'd also need to know an example of `$url`. Is it `http://www.example.com` or just `www.example.com`? There could be a difference. — Funk Forty Niner, Feb 23 '16 at 14:49
The sign is *probably* checking for cookies and headers (valid user-agent) — Derek Pollard, Feb 23 '16 at 15:29
Which means they *probably* don't want you scrapping their site — Derek Pollard, Feb 23 '16 at 15:29
Yes, I've asked them - site owner (long story short but the company is a supplier of ours and we're using a scrapper because they don't have an API yet) whether they've changed anything recently and they said they didn't. — WagnerMatosUK, Feb 23 '16 at 15:31
@divix Could you please point me in the direction on how I can do this? — WagnerMatosUK, Feb 23 '16 at 15:47
have a look here: http://stackoverflow.com/questions/8115683/php-curl-custom-headers and read about: CURLOPT_COOKIE — divix, Feb 23 '16 at 17:36

Getting a 404 while using PHP Curl but can visit the url

0 Answers0