Passing method as parameter in Python (Scrapy) - syntax

Question

I'm new to Python with some Java/C# background. I encountered the callback syntax in Scrapy tutorial and an unexpected syntax along, which I would like to understand.

In the last line of below code parse_articles_follow_next_page is a method and per my imagination I would expect a response parameter is passed there in parentheses like: yield scrapy.Request(url, self.parse_articles_follow_next_page(someresponseobject))

What sort of Python syntax is applied there so that it goes without parentheses and passing parameter, where could I read more about it?

def parse_articles_follow_next_page(self, response):
    for article in response.xpath("//article"):
        item = ArticleItem()
        #... extract article data here
        yield item

    next_page = response.css("ul.navigation > li.next-page > a::attr('href')")
    if next_page:
        url = response.urljoin(next_page[0].extract())
        yield scrapy.Request(url, self.parse_articles_follow_next_page)

score 1 · Accepted Answer · edited May 23 '17 at 12:03

1

From the Scrapy docs, one of the parameters to Request can be a callback:

callback (callable) – the function that will be called with the response of this request (once its downloaded) as its first parameter.

So you are passing a function. The response parameter will be passed to that function once it is known, by Scrapy's callback mechanism.

Read about passing functions as arguments: python function as a function argument?

Or callbacks in general: https://en.wikipedia.org/wiki/Callback_%28computer_programming%29

edited May 23 '17 at 12:03

Community

1
1

answered Oct 13 '15 at 14:33

user3468054

610
4
11

great, thanks - I might not use it again shortly outside of scrapy - but it is good to know – Turo Oct 13 '15 at 14:46

Passing method as parameter in Python (Scrapy) - syntax

1 Answers1