How to implement a website's cookies scraper?

Question

I need to create a script that make an HTTP request and simulate browser's behavior when it comes to cookie management. That means that it has access to all the cookies set by that 'page' (So server-set cookies, async client-set cookies, cookies of every domain.)

I mean, inspecting the page using a console, i can see all the cookies from all domains on that page.

I am figuring out a way to do that from my code. I am almost sure i have to somehow run page's javascript and to emulate the DOM behavior (thought to use jsDom in a node.js script).

But, but. Still have many doubts. Any suggestion on how i can intercept all cookies by simulating browser behavior ?

Any suggestion about how to implement it would be really appreciated.

for those who have time
I had a weird idea about how to implement it: Would it make sense to overload the prototype of the function of the xhr object that handles HTTP response (i mean making something like: this), to check all the client-loaded cookies ?
I need it to work on any page, even those which don't use the native XMLHttpRequest object. Any suggestion around this ?

Cameron Tinker · Answer 1 · 2016-02-12T17:52:09.097

Browsers just send HTTP requests to the server and you don't necessarily need a DOM loaded. If you can reverse engineer the requests that you want to send, you can easily mock the behavior of a web page or workflow. When you send a request with cURL from PHP, you'll need to store the cookies in a cookie jar to maintain cookies across requests. Something like this should get you started:

function load($url, $postData = array())
{
    $useragent = "User-Agent: Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/48.0.2564.109 Safari/537.36";

    $curl = curl_init();
    curl_setopt($curl, CURLOPT_URL, $url);
    curl_setopt($curl, CURLOPT_RETURNTRANSFER, TRUE);
    curl_setopt($curl, CURLOPT_HEADER, FALSE);
    curl_setopt($curl, CURLOPT_SSL_VERIFYPEER, FALSE);
    curl_setopt($curl, CURLOPT_SSL_VERIFYHOST, FALSE);
    curl_setopt($curl, CURLOPT_ENCODING, 'UTF-8');
    curl_setopt($curl, CURLOPT_USERAGENT, $useragent);
    curl_setopt($curl, CURLOPT_POST, !empty($postData));
    if(!empty($postData)) curl_setopt($curl, CURLOPT_POSTFIELDS, $postData);
    curl_setopt($curl, CURLOPT_COOKIEFILE, $cookieFile);
    curl_setopt($curl, CURLOPT_COOKIEJAR, $cookieFile);
    curl_setopt($curl, CURLOPT_FOLLOWLOCATION, TRUE);
    $page = curl_exec ($curl);
    curl_close ($curl);

    return $page;
}

If you want to run a headless browser that maintains cookies in a cookie jar, I would recommend something like PhantomJS. You can then load pages and execute code within a page's context:

"use strict";
var page = require('webpage').create();

page.onConsoleMessage = function(msg) {
    console.log(msg);
};

page.open("http://phantomjs.org/", function(status) {
    if (status === "success") {
        page.includeJs("http://ajax.googleapis.com/ajax/libs/jquery/1.6.1/jquery.min.js", function() {
            page.evaluate(function() {
                console.log("$(\".explanation\").text() -> " + $(".explanation").text());
            });
            phantom.exit(0);
        });
    } else {
      phantom.exit(1);
    }
});

The code can be found on Github: https://github.com/ariya/phantomjs/blob/master/examples/phantomwebintro.js

But javascript can also set cookies, doesn't it ? In this case i need to emulate DOM behavior — Morrisda, Feb 12 '16 at 17:04
Yes, JavaScript can set cookies. However, you could reverse engineer which cookies are set and set them manually on a request. I'm just pointing out that spinning up a browser like environment for scraping can be expensive and not very scalable. — Cameron Tinker, Feb 12 '16 at 17:07
I honestly don't need it to be scalable, but precise. Any advise on how to implement your solution ? (reverse engeneering cookie setting) — Morrisda, Feb 12 '16 at 17:15

score 0 · Answer 2 · answered Mar 09 '16 at 21:01

After some researches, i got into these conclusions:

Cookies are set by server, setting response's headers to client's requests, and from javascript on the client. Cookies set by the server are easy to be detected when they are set. But client side cookies are not so easy to be detected, because you must emulate the dom and execute the javascript of the page. So, if you want to check all the cookies that are set into a website, the only way is to run the javascript and emulate the DOM and then check for each async request sent.

I suggest you phantom.js or simply using the Chrome Remote Debugging protocol (simply get the cookies for google chrome, navigating as you were the real-world user)

How to implement a website's cookies scraper?

2 Answers2