0

So I'm trying to get into a habit of writing unit tests for my project in scrapy, but the thing is, I'm not sure how to go about this. For example, in my self.parse(), I always yield the item rather than return the item, so whenever I call a self.parse(), it returns a generator object. What I tried doing what generating a fake response as shown here:

Scrapy Unit Testing

It seems to work if you are returning an item, but what about an item that is being yielded? How do you test the items that are being yielded?

Community
  • 1
  • 1
NewbAndroider
  • 5
  • 4

1 Answers1

1

Simply consume generator into a list:

expected = MyItem({'foo': 'bar'})
item = list(spider.parse(response))
assert item == expected
Granitosaurus
  • 20,530
  • 5
  • 57
  • 82
  • @NewbAndroider if this solves your questions feel don't forget click "accept answer" button to the left of this post :) – Granitosaurus Apr 24 '17 at 21:32
  • What is there are multiple yield statements in my spider? – Kishan Mehta Mar 20 '19 at 15:37
  • 2
    @Kishan then it gets really tricky. I actually wrote a test-framework for scrapy called `scrapy-test` in still in early stages but you can find it on [pypi](https://pypi.org/project/scrapy-test/) – Granitosaurus Mar 21 '19 at 16:11