Python Scrapy not always downloading data from website

Question

Scrapy is used to parse an html page. My question is why sometimes scrapy returns the response I want, but sometimes does not return a response. Is it my fault? Here's my parsing function:

class AmazonSpider(BaseSpider):
    name = "amazon"
    allowed_domains = ["amazon.org"]
    start_urls = [
       "http://www.amazon.com/s?rh=n%3A283155%2Cp_n_feature_browse-bin%3A2656020011"
   ]

def parse(self, response):
            sel = Selector(response)
            sites = sel.xpath('//div[contains(@class, "result")]')
            items = []
            titles = {'titles': sites[0].xpath('//a[@class="title"]/text()').extract()}
            for title in titles['titles']:
                item = AmazonScrapyItem()
                item['title'] = title
                items.append(item)
            return items

Could you include the log messages of a run where you don't get the response? — R. Max, Nov 30 '13 at 05:46
Hello. Do you have any new information about it? I have similar issue http://stackoverflow.com/questions/20723371/scrapy-how-to-debug-scrapy-lost-requests — Nikolai Golub, Dec 21 '13 at 20:47
What I did was check if the titles are empty. If leeks are empty again request to the same link that I take from respinse.url . Pretty dumb solution, but it works. — Krasimir, Jan 03 '14 at 11:36
@Krasimir would you consider adding a response that shortly describes the solution you chose? — Gallaecio, Feb 07 '19 at 15:06

score 0 · Answer 1 · answered Jan 31 '14 at 11:57

I believe you are just not using the most adequate XPath expression.

Amazon's HTML is kinda messy, not very uniform and therefore not very easy to parse. But after some experimenting I could extract all the 12 titles of a couple of search results with the following parse function:

def parse(self, response):
    sel = Selector(response)
    p = sel.xpath('//div[@class="data"]/h3/a')
    titles = p.xpath('span/text()').extract() + p.xpath('text()').extract()
    items = []
    for title in titles:
        item = AmazonScrapyItem()
        item['title'] = title
        items.append(item)
    return items

If you care about the actual order of the results the above code might not be appropriate but I believe that is not the case.

Python Scrapy not always downloading data from website

1 Answers1

Linked