0

I'm trying to scrape product information from a webpage, using scrapy. this is my webpage

I saw the following posts :

selenium with scrapy for dynamic page

Scraping dynamic content using python-Scrapy

And many others ،then wrote the below code:

import scrapy
from scrapy_splash import SplashRequest

class filmnet_Spider(scrapy.Spider):
    name = 'filmnet'
    start_urls = {'http://filmnet.ir/'}

    DOWNLOADER_MIDDLEWARES = {
    'filmnet_Spider.SplashCookiesMiddleware': 723,
    'filmnet_Spider.SplashMiddleware': 725,
    'scrapy.downloadermiddlewares.httpcompression.HttpCompressionMiddleware': 810,
}

    SPIDER_MIDDLEWARES = {
    'filmnet_Spider.SplashDeduplicateArgsMiddleware': 100,
}
    DUPEFILTER_CLASS = 'filmnet_Spider.SplashAwareDupeFilter'
HTTPCACHE_STORAGE = 'filmnet_Spider.SplashAwareFSCacheStorage'

    def start_requests(self):
        for url in self.start_urls:
            yield scrapy.Request(url, self.parse, meta={
                'splash': {
                'endpoint': 'render.html',
                'args': {'wait': 0.5}
                }
            })

    def parse(self, response):

        for filmnetscrap in self.start_urls:

            poster = filmnetscrap.xpath('//div[@class="verticalImage organizer"]//img/@src').extract()
            print poster

I also wrote the setting file that included :

SPLASH_URL = 'http://localhost:8050/'

But it dosen't work

0 Answers0