use scrapy parse function to parse a specific url

Question

I have a scrapy crawler which works fine. I now want to use its 'parse' function to parse a given url. While there exists a command line utility to do so for a single url using command:

scrapy parse <options> <url>

But I want to do this inside my python code (and no starting a new process for every url is not an option)

From what I figure what I need for this is essentially a way to create Response given a url. Since the response that scrapy takes is not the same as HTTPResponse, I am not sure how to get that response given a url.

I did find a method make_reqests_from_url which does the obvious, but I am not sure how to get from scrapy Request to scrapy response, which I can pass to the parse function.

Answer 1

Managed to figure it out.

Essentially, I just needed to pass the response body, url and the scrapy request to create the response object.

bs = BaseSpider('some')
head = 'www.mywebsite.com'
httpcon = httplib.HTTPConnection(head)
tail = '/mypage.html'
httpcon.request('GET',tail)
sreq = bs.make_requests_from_url(link)
sresp = TextResponse(url=link,status=200,body=httpcon.getresponse(),encoding='utf-8')

Answer 2

A quick kludge (with pieces from here and here) in case, unlike for OP, subprocess is an option.

import subprocess
bashCommand = "Scrapy fetch http://www.testsite.com/testpage.html"
process = subprocess.Popen(bashCommand.split(), stdout=subprocess.PIPE, stderr=subprocess.PIPE)
page, scrapy_meta_info = process.communicate()