3

I have posted a similar question before. I was trying to scrape a web page using the following approach

import requests

url = 'https://www.zameen.com/'
res = requests.get(url)
data = res.text
print(data)

Its response says I am either a BOT or Javascript is not enabled. So, I have check but Javascript is enabled. So I tried another approach of using fake user agent with the following code

from fake_useragent import UserAgent
headers = {}
headers['User-Agent'] = str(ua.chrome)
web_page = requests.get(url,headers=headers)
print(web_page.content)

Response:

b'<!DOCTYPE html>\n\n\t\n\n\t\n\t\n\t\n\n\t\n\t\n\n\t\n\t\n\t\n\n<head>\n<META NAME="ROBOTS" CONTENT="NOINDEX, NOFOLLOW">\n<meta http-equiv="cache-control" content="max-age=0" />\n<meta http-equiv="cache-control" content="no-cache" />\n<meta http-equiv="expires" content="0" />\n<meta http-equiv="expires" content="Tue, 01 Jan 1980 1:00:00 GMT" />\n<meta http-equiv="pragma" content="no-cache" />\n<meta http-equiv="refresh" content="10; url=/distil_r_captcha.html?Ref=/&amp;distil_RID=053235A2-0030-11E7-8429-B03805AB611E&amp;distil_TID=20170303163950" />\n<script type="text/javascript">\n\t(function(window){\n\t\ttry {\n\t\t\tif (typeof sessionStorage !== \'undefined\'){\n\t\t\t\tsessionStorage.setItem(\'distil_referrer\', document.referrer);\n\t\t\t}\n\t\t} catch (e){}\n\t})(window);\n</script>\n<script type="text/javascript" src="/ga368490.js" defer></script><style type="text/css">#d__fFH{position:absolute;top:-5000px;left:-5000px}#d__fF{font-family:serif;font-size:200px;visibility:hidden}#caexxxzxycbzutyvy{display:none!important}</style></head>\n<body>\n<div id="distil_ident_block">&nbsp;</div>\n</body>\n</html>\n'

It again detected me as a robot. So I checked whether I can fetch the data from the website or not. Then I used robotparser from urllib

from urllib import robotparser

req = robotparser.RobotFileParser()
req.set_url(url)
req.read()
print(req.can_fetch('*','https://www.zameen.com/'))

Returns:

TRUE # Means I can fetch the data from the website.

Is there any way to get the data from this web page ? Thanks

Community
  • 1
  • 1
muazfaiz
  • 4,611
  • 14
  • 50
  • 88
  • Please check this answer: http://stackoverflow.com/questions/8049520/web-scraping-javascript-page-with-python – foobar Mar 03 '17 at 16:55
  • I am not sure what's happening. I tried using mechanize and setting robots_handle to false but for some reason it was giving 405 Error. Same with requests i am getting 405 error – Shashank Mar 03 '17 at 16:59
  • I should just note that the website in question wishes not to be visited by bots, via the meta tag found in the response: `` –  Mar 03 '17 at 17:58

1 Answers1

2

You can use BeautifulSoup and a Selenium driver for this. I obtained success getting the page source from the URL you provided with this:

from selenium import webdriver
from bs4 import BeautifulSoup

driver = webdriver.Firefox() # Could be any other browser you have the drivers for
driver.get('https://zameen.com')
html = driver.page_source
code = BeautifulSoup(html, 'html5lib')
print code

Just don't forget to install bs4 and Selenium with:

pip install bs4

pip install selenium
halfer
  • 19,824
  • 17
  • 99
  • 186
Dico
  • 320
  • 4
  • 7