1

I wanna scrape images from yandex with xpath.

Here's an example link : https://yandex.ru/images/search?from=tabbar&text=B%20E%20N%20N%20Y%20-Benny_Electronic

Here's my xpath, seems to work in chropath extension (if you change the number of the last div it will switch to next image)

//div[@class="serp-list serp-list_type_search serp-list_unique_yes serp-list_rum_yes serp-list_justifier_yes serp-controller__list counter__reqid clearfix i-bem serp-list_js_inited"]/div[1]'

Problem is that response doesn't return anything when I'm using scrapy. Here's my code :

import scrapy
from letmerun.folder_music import Utility

class YandexSpider(scrapy.Spider):
    name = 'yandex'
    start_urls = Utility.artworks_urls
    img_links = []

def parse(self, response):
    links = response.xpath('//div[@class="serp-list serp-list_type_search serp-list_unique_yes serp-list_rum_yes serp-list_justifier_yes serp-controller__list counter__reqid clearfix i-bem serp-list_js_inited"]/div[1]')
    print(links)
Mikhail Aleksandrovskiy
  • 51
  • 3
  • Try using the interactive scrapy shell (```scrapy shell``` in a terminal, ```fetch``` method) to request that page. You'll see that it doesn't load the content you're looking for, because it uses JavaScript to dynamically load that content. You need to either load that content manually, or find a package to do that for you with scrapy. – Chillie Nov 06 '20 at 08:25
  • 1
    Does this answer your question? [selenium with scrapy for dynamic page](https://stackoverflow.com/questions/17975471/selenium-with-scrapy-for-dynamic-page) – Chillie Nov 06 '20 at 08:26
  • [This](https://stackoverflow.com/questions/39057169/specyfic-xpath-doesnt-work-in-pycharm-python-selenium-webdriver) might resolve your issue. It worked for me in Selenium. – Cyrus Jul 31 '22 at 22:18

0 Answers0