How to scrape page requiring cookies and javascript in PHP

Question

Is there an easy way to emulate cookies and javascript with a php script scraping a web page requiring it?

The current response shows:

<body><noscript>This site requires JavaScript and Cookies to be enabled. Please change your browser settings or upgrade your browser.</noscript></body>

I put this in the code and it made no difference:

$strCookie = 'PHPSESSID=' . $_COOKIE['PHPSESSID'] . '; path=/';
curl_setopt( $ch, CURLOPT_COOKIE, $strCookie );

Is there a specific reason why you would want to do it at the server end rather than the client end? — francisco.preller, Mar 09 '12 at 08:11
The script is parsing to create an rss feed, to then become a twitter feed and the underlying site is happy with the arrangement too. — Adrian33, Mar 09 '12 at 10:33

score 2 · Accepted Answer · edited May 23 '17 at 12:23

HTML inside the <noscript> </noscript> will only get displayed if Javascript is disabled. So this is the test for whether Javascript is enabled.

There is jQuery-cookie-plugin for reading, writing and deleting cookies. But it will not work if Javascript is disabled.

For checking if cookie is enabled with PHP code, you need to set the cookie and read the cookie with another request. This post have some guides on it.

score 0 · Answer 2 · answered Mar 09 '12 at 07:55

0

Is there an easy way to emulate cookies and javascript with a php script scraping a web page requiring it?

No.

answered Mar 09 '12 at 07:55

Your Common Sense

156,878
40
214
345

How to scrape page requiring cookies and javascript in PHP

2 Answers2