13

I'm trying to crawl a website which return an error message that your js is disabled and you might be a bot. I tried to see same behaviour in web browser and yes the same response, however if JavaScript is enabled it will not affect the original response, I mean original response is not dependent on JS.

So I was thinking if I can tell the web/http server that my JS is enabled and I'm not a BOT. is this possible in Python requests library, or any other python library for that matter?

And yeah I've set the User-Agent header, even all other headers, like host, language, connection, etc

bakar
  • 1,152
  • 2
  • 11
  • 20
  • Is the page executing some script that would tell it JavaScript is disabled? Something that no number of headers would bypass. – Mike McMahon Oct 04 '15 at 23:42
  • @MikeMcMahon: http://delhi.quikr.com/Bikes-Scooters/w264 if I disable js it returns some message. – bakar Oct 05 '15 at 05:30

1 Answers1

19

If the site is just checking whether javascript can be executed or not through executing some js, use selenium to get the page, and then use BeautifulSoup to parse the page that selenium got.

from bs4 import BeautifulSoup
from selenium import webdriver
driver = webdriver.Firefox()
driver.get('http://your-site/url')

html = driver.page_source
soup = BeautifulSoup(html)
...
Flickerlight
  • 904
  • 8
  • 18