1

I am trying to scrape data from magicbricks.com But when i try to change the page by manually clicking on the second page at the bottom of the page, the page link remains the same. And i get the same data. How can i load the remaining pages.

For Example: This is the link of first page.

https://www.magicbricks.com/property-for-sale/residential-real-estate?bedroom=1,2,3,4,5,%3E5&proptype=Multistorey-Apartment,Builder-Floor-Apartment,Penthouse,Studio-Apartment,Residential-House,Villa,Residential-Plot&cityName=Mumbai

Link of second page is the same only content of page changes

https://www.magicbricks.com/property-for-sale/residential-real-estate?bedroom=1,2,3,4,5,%3E5&proptype=Multistorey-Apartment,Builder-Floor-Apartment,Penthouse,Studio-Apartment,Residential-House,Villa,Residential-Plot&cityName=Mumbai

import pandas as pd
from pandas import ExcelWriter
import requests,re,csv
from bs4 import BeautifulSoup

for i in range(1,5):      # Number of pages plus one 

   url = "https://www.magicbricks.com/property-for-sale/residential- 
   real-estate?bedroom=1,2,3,4,5,%3E5&proptype=Multistorey- 
   Apartment,Builder-Floor-Apartment,Penthouse,Studio- 
   Apartment,Residential-House,Villa,Residential- 
   Plot&cityName=Mumbai".format(i);

   r = requests.get(url)
   soup = BeautifulSoup(r.content)

I want to scrape 500 entries of this website

DisappointedByUnaccountableMod
  • 6,656
  • 4
  • 18
  • 22
Karan Dumbre
  • 33
  • 1
  • 8
  • use web data scrapper tool or extension for google chrome. – Dhruv Raval Aug 22 '18 at 04:52
  • @Karan, unfortunately, i cannot see the website, but I will be some kind of list of things you want to scrap, correct? And this list has some kind of "next" button where the content changes but the URL does not. This would guess that the "next" button is just an ajax-call; only the affected content gets updated. If this guess is correct, I would recommend to use scrapy and write a crawler and have a look at [this](https://stackoverflow.com/questions/39486224/scraping-ajax-page-with-scrapy) – Marvin Taschenberger Aug 22 '18 at 05:51
  • Ok. But i still cant figure out how to navigate through ajax pages – Karan Dumbre Aug 22 '18 at 09:44
  • – Karan Dumbre Aug 22 '18 at 09:53
  • It has a variable data-position="3" , which i can find unique. I want data of 500 entries. – Karan Dumbre Aug 22 '18 at 09:54

1 Answers1

0

try to use selenium for this browser.execute_script("window.scrollTo(0, document.body.scrollHeight);") and this piece od code to scroll

swaraj
  • 26