Storing scraped data into the database sqlite

Question

I have used scarpy to scrap some text from a website. But I am not quite sure how to store them in sqlite?Can anyone help me with the code?

You need to define a Pipeline like suggested [here](http://stackoverflow.com/questions/3261858/does-anyone-have-example-code-for-a-sqlite-pipeline-in-scrapy), or use an item exporter, like [this](https://github.com/RockyZ/Scrapy-sqlite-item-exporter) one. Also see this [scrapy-dblite](https://pypi.python.org/pypi/scrapy-dblite/0.2.5) library. — alecxe, Jan 14 '14 at 06:35

score 3 · Accepted Answer · answered Jan 14 '14 at 06:49

while you can find some examples that are using blocking operations to interact with the database it is worth noting that scrapy is built on top of twisted library, meaning that in its core there is only a single thread with a single loop for all operations, so when you do something like:

self.cursor.execute(...)

the entire system is waiting for a response from the database, including http requests that are waiting to be executed etc.

having said that, I suggest you'll check this code snippet https://github.com/riteshk/sc/blob/master/scraper/pipelines.py

using twisted.enterprise.adbapi.ConnectionPool is a little more complex than a simple blocking database access code but it plays well with the way scrapy uses io operations

Storing scraped data into the database sqlite

1 Answers1