Get The Redirect URL that has json Data Using CURL

Question

I have URL like this www.example.com/abc/xyz and when it is loaded it turns into the different url like:

www.example.com/abc/xyz#facet=c_State+s_FL+p14_2+r14_5g6tVndJ+n10_2+x10_2BLnMX+b5_0+h5_0+g3_0+f3_0+v_No Preference+t8_0+a30_0+u_0+k_0+q_0+w_false+j_Q+e_1+i_2

I just want the second url ..so that I can use that parameters to send the json data using curl.

The second url is requesting the json data so that I am not able to track it.I want that data using first URL.

Here is my code:

$url = "http://www.lennar.com/New-Homes/Florida/Tampa";
$ch = curl_init();
curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, false);
curl_setopt($ch, CURLOPT_FOLLOWLOCATION , false);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
curl_setopt($ch, CURLOPT_HTTPHEADER, array(
    'Content-Type: application/json',
    )
);
curl_setopt($ch, CURLOPT_URL, $url);
$result = curl_exec($ch);
curl_close($ch);
var_dump($result);

and My output is :

string '



    <!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">



    <html xmlns="http://www.w3.org/1999/xhtml" lang="en-us" xml:lang="en-us" xmlns:fb="http://ogp.me/ns/fb#" xmlns:og="http://opengraphprotocol.org/schema/">

    <head id="Head1"><meta name="keywords" content="quality, new homes, home builder" />

    <meta name="description" content="Lennar Homes: If a home builder provided everything you want and everything you need, and by doing so, was a'... (length=165671)

Do you want curl to prevent the redirection to the second url ? — Himal, Sep 26 '14 at 10:11
NO..I just want the second URL. But it is not getting in curl_getinfo($ch, CURLINFO_EFFECTIVE_URL); — chirag, Sep 26 '14 at 10:19
Yes, that's what i meant. to get the second url you have to prevent the redirection and get the second url from the `Location` header.is that what you want ? — Himal, Sep 26 '14 at 10:21
Have you tried setting `CURLOPT_FOLLOWLOCATION` to 'false' ? — Himal, Sep 26 '14 at 10:22
I am using curl and I cant prevent the redirection ..Because on the base of the second url's data I will make another request.I just want second url anyhow my friend. — chirag, Sep 26 '14 at 10:22
So, you are going to make another separate curl request to the second url right ? — Himal, Sep 26 '14 at 10:24
Yes I have tried CURLOPT_FOLLOWLOCATION to false..But it is not working. — chirag, Sep 26 '14 at 10:25
What do you mean not working ? Is it still redirecting to the second url ? have you dumped the response headers ? can't you see the location header ? — Himal, Sep 26 '14 at 10:27
Let us [continue this discussion in chat](http://chat.stackoverflow.com/rooms/61968/discussion-between-chirag-and-himal). — chirag, Sep 26 '14 at 10:28
@chirag You have to post your code. Otherwise no one can help you — hindmost, Sep 26 '14 at 10:44

score 1 · Answer 1 · edited May 23 '17 at 11:57

The so called "second URL" that you desire to get is generated by JavaScript.

Notice that you are not redirected by the first URL to the second one. The JS generated under the first URL after page load just adds some data to the first URL.

Neither curl, nor wget or file_get_contents will get you what you want, since none of those parses/executes JavaScript code.

If you want to simulate creating the second URL, you could try inspecting JS on that page (I'd say bad idea, needs loads of work and is vulerable to external logic changes) or try looking at SO: Get sourcecode after javascript execution with curl, and then using JS engine to get the second URL.

If you want to play with simulating the URL, I'd look at var facetContextJSON present in loaded HTML - it's some kind of config JSON, that you can get by cURL, regexp+parse it and figure out how to build the URL you actually need.

score 0 · Accepted Answer · answered Oct 01 '14 at 07:48

I have found the way to get the json parameters which I was not able to get through curl. Here is the solution :

$.get(
                                        'http://www.example.com',
                                        function(response) {
                                           //your response as html of that page

                                        });

From the above script I got whole html of that page and also get the json data.Then by using the string operations I found exact variables which I wanted for my next url.

Get The Redirect URL that has json Data Using CURL

2 Answers2