0

So I start my scrapy shell with

scrapy shell 'https://www.amazon.com/s?k=tomatoes&ref=nb_sb_noss_1'

And I am trying to scrape the title of the products so I enter

response.xpath('//span[@class="a-size-base-plus a-color-base a-text-normal"]').getall()

and get: []

And when I tried it in CSS with

response.css("span.a-size-base-plus a-color-base a-text-normal").getall()

I still get: []

I don't understand why it isn't finding the element even though I am copying and pasting the tags and classes from the site.

I have also tried just writing a-size-base-plus for the classes in XPath and CSS but I still get nothing

Vin
  • 968
  • 2
  • 10
  • 22
BirdBud
  • 143
  • 1
  • 15

1 Answers1

1

You need to replace the spaces between the classes with dots if you want to match multiple classes on elements:

response.css("span.a-size-base-plus.a-color-base.a-text-normal").getall()

instead of:

response.css("span.a-size-base-plus a-color-base a-text-normal").getall()

Your selector with spaces says something like: Give me all a-text-normal elements inside a-color-base elements inside span elements with a class of a-size-base-plus. This is of course not what you want.

I've also had to set a user agent to get the correct results. See this answer on how to set the user agent using scrapy shell.

5eb
  • 14,798
  • 5
  • 21
  • 65