0

I had one question , may be its wierd , but wanna know it,

I tried to access facebook with URl www.facebook.com through scrapy. I had given it in start_url. After running i got the response as http://m.facebook.com/?refsrc=http%3A%2F%2Fwww.facebook.com%2F&_rdr , when we open this url in browser,i can expect this is mobile view for facebook. So why the response is mobile view and not the general view that we are able to see when opening on a desktop.

Thanks in advance...................

Shiva Krishna Bavandla
  • 25,548
  • 75
  • 193
  • 313

1 Answers1

0

There is a global setting for that: USER_AGENT

UPDATE :

You know, maybe dealing with the mobile version is an advantage after all. Other sites redirect browsers to other pages when no javascript can be executed:

<noscript> <meta http-equiv="refresh" content="0; URL=/homedepot?_fb_noscript=1" /> </noscript>

Dealing with a no js version or a mobile version of the site means less sizes of the pages and less additional info on the page - so the html will not change much over times, and your xpath queries will still work.

In that case just disable JS in Firefox or set a different User-Agent in it to get the same pages scrapy gets. Here are more hints on how to use Firefox for testing scrapy: Using Firefox for scraping

warvariuc
  • 57,116
  • 41
  • 173
  • 227