2

Hi all I have seen lots of questions regarding this. I know that javascript dynamic page will rendered using scrapyjs or webdriver like selenium or phantomjs. webdriverkit is bit slow. I want somebody to guide me in this link

Price info before view deal button. I don't know which js is executing for this to use splash, scrapyjs can someone help me for this link.

thanks in advance.

EDIT

as per andres reply i have recreated XHR request. when we enter the XHR request url in browser window since it is a GET method if first hit i got partial json output. if we hit reload next time it loads more data that seems weired. can anyone help me in this. thanks in advance

Sabeena
  • 85
  • 12

1 Answers1

1

When you request this URL:

http://ar.trivago.com/?iPathId=38715&iGeoDistanceItem=47160&aDateRange%5Barr%5D=2016-01-01&aDateRange%5Bdep%5D=2016-01-02&iRoomType=7&tgs=4716002&aHotelTestClassifier=&aPriceRange%5Bfrom%5D=0&aPriceRange%5Bto%5D=0&iIncludeAll=0&iGeoDistanceLimit=20000&aPartner=&iViewType=0&bIsSeoPage=false&bIsSitemap=false&

An XHR request is made to:

http://ar.trivago.com/search/region?iPathId=38715&bDispMoreFilter=false&iSlideOutItem=47160&aDateRange%5Barr%5D=2016-01-01&aDateRange%5Bdep%5D=2016-01-02&aCategoryRange=0%2C1%2C2%2C3%2C4%2C5&iRoomType=7&sOrderBy=relevance%20desc&aPartner=&aOverallLiking=1%2C2%2C3%2C4%2C5&iGeoDistanceLimit=20000&iOffset=0&iLimit=25&iIncludeAll=0&bTopDealsOnly=false&iViewType=0&aPriceRange%5Bfrom%5D=0&aPriceRange%5Bto%5D=0&iGeoDistanceItem=47160&aGeoCode%5Blng%5D=-0.1589&aGeoCode%5Blat%5D=51.513802&bIsSeoPage=false&mgo=false&bHotelTestContext=false&th=false&aHotelTestClassifier=&bSharedRooms=false&bIsSitemap=false&rp=&sSemKeywordInfo=&tgs=4716002&bRecommendedItem=false&iFilterTab=0&&_=1446673248317

Where you can find these values (in JSON format):

enter image description here

Which are the ones showed here:

enter image description here

So I think you don't need any ScrapyJS nor PhantomJS to scrape that information. Just understand where is it getting the information from and scrape the endpoint, directly.

Andrés Pérez-Albela H.
  • 4,003
  • 1
  • 18
  • 29
  • Thank you so much for your reply it is more helpful. i will recreate the xhr reguest using scrapy form request. but in chrome developer tool i didnt find any XHR request. can you tell me how to get this while executing so that i can compare the requests with other hotels. thank you once more. – Sabeena Nov 05 '15 at 05:34
  • @Sabeena Good to know that my answer helped you! If this answer meets your expectations, please set it as the answer for your question by clicking on the check mark. It's my pleasure. – Andrés Pérez-Albela H. Nov 05 '15 at 05:36
  • 1
    hi Andres, thaank you so much for your help. yesterday i really stuck in this. i have mistakenly unchecked the preserve log in network tab developer tool that is the reason i didnt see the XHR request. once you confirmed i checked and foud this. thank you so much for your help and support. – Sabeena Nov 05 '15 at 06:01
  • Hi andres, sorry for troubling you again. i have noted while clicking after enter the hotelname wen we hit search the following XHR excuting Request URL:http://ar.trivago.com/search/region?iPathId=38715&iGeoDistanceItem=47160&aDateRange%5Barr%5D=2015-11-12&aDateRange%5Bdep%5D=2015-11-13&iRoomType=7&tgs=4716002&aHotelTestClassifier=&aPriceRange%5Bfrom%5D=0&aPriceRange%5Bto%5D=0&iIncludeAll=0&iGeoDistanceLimit=20000&aPartner=&iViewType=0&bIsSeoPage=false&bIsSitemap=false&&_=1446750979673 – Sabeena Nov 05 '15 at 19:37
  • the developer tool response for the above xhr is large seems good. but through scrapy i have reproduced the above request. got bit less response moreover data is lossed. – Sabeena Nov 05 '15 at 19:41
  • yield scrapy.Request(url, dont_filter = 'TRUE', callback=self.parse_data, errback=self.parse_error) this is the request i am recreating using scrapy. url is wtevever in above comment. – Sabeena Nov 05 '15 at 19:43
  • @Sabeena I receive the same response both in the browser and in Scrapy. We can't use this comments as a chat (SO rules), so if you want any more help I would recommend to continue our conversation in other place. Let me know by twitter. – Andrés Pérez-Albela H. Nov 05 '15 at 19:49
  • @ Andrés Pérez-Albela H can you please reply i m getting partial response which dont hav price details. thanks in advance – Sabeena Nov 06 '15 at 15:11
  • I don't know how to parse that JSON in scrapy – Aminah Nuraini Jan 26 '18 at 20:12