I'm trying to scrape product information from a webpage, using scrapy. this is my webpage
I saw the following posts :
selenium with scrapy for dynamic page
Scraping dynamic content using python-Scrapy
And many others ،then wrote the below code:
import scrapy
from scrapy_splash import SplashRequest
class filmnet_Spider(scrapy.Spider):
name = 'filmnet'
start_urls = {'http://filmnet.ir/'}
DOWNLOADER_MIDDLEWARES = {
'filmnet_Spider.SplashCookiesMiddleware': 723,
'filmnet_Spider.SplashMiddleware': 725,
'scrapy.downloadermiddlewares.httpcompression.HttpCompressionMiddleware': 810,
}
SPIDER_MIDDLEWARES = {
'filmnet_Spider.SplashDeduplicateArgsMiddleware': 100,
}
DUPEFILTER_CLASS = 'filmnet_Spider.SplashAwareDupeFilter'
HTTPCACHE_STORAGE = 'filmnet_Spider.SplashAwareFSCacheStorage'
def start_requests(self):
for url in self.start_urls:
yield scrapy.Request(url, self.parse, meta={
'splash': {
'endpoint': 'render.html',
'args': {'wait': 0.5}
}
})
def parse(self, response):
for filmnetscrap in self.start_urls:
poster = filmnetscrap.xpath('//div[@class="verticalImage organizer"]//img/@src').extract()
print poster
I also wrote the setting file that included :
SPLASH_URL = 'http://localhost:8050/'
But it dosen't work