0

I am trying to get the price texts in potterybarn in scrapy shell. I used scrapy shell "https://www.potterybarnkids.com/shop/easter/easter-shop-all-baskets/" then trying to get the price inside span class="price-state price-sale" is there a way to extract entire text inside span with going into each span inside it?

I tried

response.xpath('//span[@class="price-state price-sale"]/text()').extract() also response.xpath('//span[@class="price-state price-sale"]//text()')[0].extract()

I need a way to extract all texts inside the selector weather it has inner spans,divs,...

I checked How can i extract only text in scrapy selector in python, also Scrapy extracting text from div in this one the answer assumes that it will contain only span children which will work in that example and this one. but is there a more general way to correctly extract all text inside children because //text() isn't working.

  • I took a quick glance at the HTML. It appears that the span elements of class "price-amount" are the ones with the price as inner text. The span elements of class "price-state price-sale" have '\n' as their inner text. – dmmfll May 18 '19 at 18:49
  • @DMfll the span contains the text "Sale $5.99 – $18.99" in html format (including \n ...) –  May 18 '19 at 18:53
  • Yes. I see. I'm using lxml to look at the elements. You will find that if you iterate over all the descendants as opposed to children of the element at `'/html/body/div[3]/main/div/div[4]/div[2]/ul/li[1]/span/span[2]'` you will get an array of `['\nSale\n', '\n', '$', '5.99', '\n', '$', '18.99']` as text items. – dmmfll May 18 '19 at 19:12

1 Answers1

1

I think there are more efficient ways, but the following xpath does the job. The string() on xpath gather text from all children nodes.

You can find more information about differences between string() and text() on this post Difference between text() and string()

prices = [
    r.xpath('string(.)').extract_first() 
    for r in response.xpath('//span[@class="price-state price-sale"]')
]

As you can see on results, one product per row. It could be cleaned with replace for example, or prices extracted using regex

>>> prices
['\n\nSale\n\n\n$5.99\n–\n\n$18.99\n', '\n\nSale\n\n\n$6...

Other option would be to do it in two steps, using text() instead of string() and cleaning data before join operation:

>>> prices = []
>>> for r in response.xpath('//span[@class="price-state price-sale"]'):
>>>     price = [p.strip() for p in r.xpath('.//text()').extract() if p.strip()]
>>>     prices.append(' '.join(price))

Results in this case are already cleaned

>>> prices
['Sale $ 5.99 – $ 18.99', 'Sale $ 6.99 – $ 18.99', 'Sale $ 6.99...
Marcos
  • 645
  • 3
  • 10