save screen shot of scrapy response

Question

Is there a way to save Scrapy response screenshot of page, I.e

scrapy shell "https://google.com" 
view(response)

I know I can save the output as HTML and view it later, but is there a way to save the output as image?

I checked this Question Scrapy Splash Screenshots?, (the most relevant one) but I get

png_bytes = base64.b64decode(response.data['png'])
Traceback (most recent call last):
  File "/usr/lib/python3.6/code.py", line 91, in runcode
    exec(code, self.locals)
  File "<console>", line 1, in <module>
AttributeError: 'HtmlResponse' object has no attribute 'data'

I assume this error is because in Question he uses Splash Request,in my case normal Request

score 0 · Answer 1 · answered Aug 26 '19 at 20:35

0

Do you mean a screenshot of the webpage or the output in your command line?

The other question you provided seems to be talking about screenshots on the pages being scraped.

The Scrapy documentation has some info on that https://docs.scrapy.org/en/latest/topics/item-pipeline.html?highlight=screenshot#take-screenshot-of-item I'm not sure if splash is required

answered Aug 26 '19 at 20:35

reg202

160
7

it doesn't work,"Could not load image yyy.png" not a PNG file – programmerwiz32 Aug 26 '19 at 20:44
Looks like splash is required but the least you can do is answer the question – reg202 Aug 26 '19 at 20:51
well, there is an option I can use but very remote from this, is to simply use headless browser (selenium) and feed it the HTML response and take screenshot there – programmerwiz32 Aug 26 '19 at 21:40

score 0 · Answer 2 · answered Aug 30 '19 at 10:53

0

Splash is the most common approach for this.

See https://splash.readthedocs.io/en/stable/api.html#render-png after you’ve read a bit about Splash in general.

answered Aug 30 '19 at 10:53

Gallaecio

3,620
2
25
64

save screen shot of scrapy response

2 Answers2