Scraping comment section with Python using Scrapy

Asked Jun 04 '17 at 15:46

Active Jun 04 '17 at 15:53

Viewed 133 times

I came across a problem while scraping the comments from this link.

Using scrapy shell, I'm wondering why I cannot see the comments in this below div section.

['<div id="eqWall" data-project-id="53481" data-is-platform-manager="" data-replies-allowed="1"></div>']

I got the above output by executing response.xpath("//section[@class='main-section pin-board-section']/div/div[@id='eqWall']").extract() using scrapy shell "link" Could anyone help me out why I cannot see and what is the solution to scrape the comments.

Thanks

edited Jun 04 '17 at 15:53

asked Jun 04 '17 at 15:46

Srivardhan Cholkar

As Andersson suggests you're trying to load javascript content. Try opening the page source (not inspect, but page source) and you'll see that this div is actually empty. – Casper Jun 05 '17 at 09:13
@Andersson after so many error I finally got the result as suggested thanks a lot :) – Srivardhan Cholkar Jun 05 '17 at 13:53
@Casper Yes using view source I can see empty. Tried spash thing, worked well. Thanks :) – Srivardhan Cholkar Jun 05 '17 at 13:54

Scraping comment section with Python using Scrapy

0 Answers0