Scrapy error : exceptions.AttributeError: 'HtmlResponse' object has no attribute 'urljoin'

Question

I have installed scrapy using pip and tried the example in the scrapy documentation.

I got the error cannot import name xmlrpc_client

After looking into a stachoverflow question here i have fixed it using

sudo pip uninstall scrapy

sudo pip install scrapy==0.24.2

but now it shows me exceptions.AttributeError: 'HtmlResponse' object has no attribute 'urljoin'

Here is my code :

import scrapy


class StackOverflowSpider(scrapy.Spider):
    name = 'stackoverflow'
    start_urls = ['https://stackoverflow.com/questions?sort=votes']

    def parse(self, response):
        for href in response.css('.question-summary h3 a::attr(href)'):
            full_url = response.urljoin(href.extract())
            yield scrapy.Request(full_url, callback=self.parse_question)

    def parse_question(self, response):
        yield {
            'title': response.css('h1 a::text').extract()[0],
            'votes': response.css('.question .vote-count-post::text').extract()[0],
            'body': response.css('.question .post-text').extract()[0],
            'tags': response.css('.question .post-tag::text').extract(),
            'link': response.url,
        }

Can anyone please help me with this!

alecxe · Accepted Answer · 2015-11-04T18:02:24.560

6

In Scrapy >=0.24.2, HtmlResponse class does not yet have urljoin() method. Use urlparse.urljoin() directly:

full_url = urlparse.urljoin(response.url, href.extract())

Don't forget to import it:

import urlparse

Note that urljoin() alias/helper was added in Scrapy 1.0, here is the related issue:

Add Response.urljoin() helper

And here is what it actually is:

from six.moves.urllib.parse import urljoin

def urljoin(self, url):
    """Join this Response's url with a possible relative url to form an
    absolute interpretation of the latter."""
    return urljoin(self.url, url)

edited Nov 04 '15 at 18:02

answered Jun 30 '15 at 18:29

alecxe

462,703
120
1,088
1,195

But how can the example shown in their website be wrong? And when i tried it with urlparse,it is giving me `urljoin() takes at least 2 arguments (1 given)` – user3437315 Jun 30 '15 at 18:34
1

@user3437315 the example is for Scrapy 1.0 while you are using 0.24.2 – alecxe Jun 30 '15 at 18:37
Thanks,But where can i get documentation for Scrapy 0.24.2 .It doesn't seem to be there on their website – user3437315 Jun 30 '15 at 18:41
@user3437315 try just uninstall scrapy and installing again. – alecxe Jun 30 '15 at 18:51
I did that.i get cannot import name xmlrpc_client error! I tried reinstalling six as well.But not working! – user3437315 Jun 30 '15 at 18:52

score 0 · Answer 2 · answered Jun 30 '15 at 18:37

The example code you're using is for scrapy 1.0. Since you've downgraded to 0.24, you need to use the urljoin from urlparse:

full_url = urljoin(response.url, href.extract())

If you click the "Scrapy 0.24 (old stable)" button above the example you'll get example code for the scrapy version you're using.

Scrapy error : exceptions.AttributeError: 'HtmlResponse' object has no attribute 'urljoin'

2 Answers2