Questions tagged [scrapy-pipeline]
218 questions
18
votes
1 answer
Scrapy: how to use items in spider and how to send items to pipelines?
I am new to scrapy and my task is simple:
For a given e-commerce website:
crawl all website pages
look for products page
If the URL point to a product page
Create an Item
Process the item to store it in a database
I created the spider but…
farhawa
- 10,120
- 16
- 49
- 91
7
votes
1 answer
Scrapy, make http request in pipeline
Assume I have an scraped item that looks like this
{
name: "Foo",
country: "US",
url: "http://..."
}
In a pipeline I want to make a GET request to the url and check some headers like content_type and status. When the headers do not meet…
DarkLeafyGreen
- 69,338
- 131
- 383
- 601
6
votes
1 answer
Closing database connection from pipeline and middleware in Scrapy
I have a Scrapy project that uses custom middleware and a custom pipeline to check and store entries in a Postgres DB. The middleware looks a bit like this:
class ExistingLinkCheckMiddleware(object):
def __init__(self):
... open…
Jamie Brown
- 1,023
- 10
- 13
5
votes
2 answers
Scrapy file download how to use custom filename
For my scrapy project I'm currently using the FilesPipeline. The downloaded files are stored with a SHA1 hash of their URLs as the file names.
[(True,
{'checksum': '2b00042f7481c7b056c4b410d28f33cf',
'path':…
Michael
- 2,436
- 1
- 36
- 57
5
votes
1 answer
Scrapy Pipelines to Seperate Folder/Files - Abstraction
I currently finalising a Scrapy project however I have quite a lengthy pipelines.py file.
I noticed that in my settings.py the pipe lines are show as follows (trimmed down):
ITEM_PIPELINES = {
'proj.pipelines.MutatorPipeline': 200,
…
Matt The Ninja
- 2,641
- 4
- 28
- 58
5
votes
1 answer
Django Relations with Scrapy how are items saved?
I just need to understand How can I detect whether scrapy saved and item in spider ? I'm fetching items from a site and after that I'm fetching comments on that item. So first I have to save the item after that I'll save comments. But when I'm…
Murat Kaya
- 1,281
- 3
- 28
- 52
5
votes
0 answers
Twisted (Scrapy) and Postgres
Im using Scrapy (aka Twisted) and also Postgres as a database.
After I while my connections seem to fill up and then my script is been stuck. I checked this with this query SELECT * FROM pg_stat_activity; and read that its caused because Postgres…
lony
- 6,733
- 11
- 60
- 92
5
votes
2 answers
scrapy - handling multiple types of items - multiple and related Django models and saving them to database in pipelines
I have the following Django models. I am not sure what is the best way to save these inter-related objects when scanned in spider to the database in Django using scrapy pipelines. Seems like scrapy pipeline was built to handle only one 'kind' of…
dowjones123
- 3,695
- 5
- 40
- 83
5
votes
1 answer
When saving scraped item and file, Scrapy inserts empty lines in output csv file
I have Scrapy (version 1.0.3) spider in which I extract both some data from web page and I also download file, like this (simplified):
def extract_data(self, response):
title = response.xpath('//html/head/title/text()').extract()[0].strip()
…
zdenulo
- 346
- 3
- 14
4
votes
1 answer
Custom Files Pipeline in Scrapy never downloads Files even though logs should all functions being accessed
I have the following custom pipeline for downloading JSON files. It was functioning fine until I need to add the __init__ function, in which I subclass the FilesPipeline class in order to add a few new properties. The pipeline takes URLs that are to…
CaffeinatedMike
- 1,537
- 2
- 25
- 57
4
votes
1 answer
Export scrapy items to different files
I'm scraping review from moocs likes this one
From there I'm getting all the course details, 5 items and another 6 items from each review itself.
This is the code I have for the course details:
def parse_reviews(self, response):
l =…
Luis Ramon Ramirez Rodriguez
- 9,591
- 27
- 102
- 181
4
votes
2 answers
Scrapy store returned items in variables to use in main script
I am quite new to Scrapy and want to try the following:
Extract some values from a webpage, store it in a variable and use it in my main script.
Therefore I followed their tutorial and changed code for my purposes:
import scrapy
from scrapy.crawler…
MaGi
- 171
- 1
- 1
- 10
4
votes
1 answer
How to download image using Scrapy?
I am newbie to scrapy. I am trying to download an image from here. I was following Official-Doc and this article.
My settings.py looks like:
BOT_NAME = 'shopclues'
SPIDER_MODULES = ['shopclues.spiders']
NEWSPIDER_MODULE =…
Prashant Prabhakar Singh
- 1,120
- 4
- 15
- 33
4
votes
2 answers
Understand the scrapy framework architecture
Recently, I've been trying to get to grips with scrapy. I feel if I had a better understanding to the architecture, I'd move a lot faster. The current, concrete problem I have this: I want to store all of the links that scrapy extracts in a…
user3185563
- 1,314
- 2
- 15
- 22
3
votes
1 answer
Scrapy: upgrade the pipeline to send items
I have a class in pipelines.py that sends and threads to my server's API:
class MyPipeline:
def process_item(self, item, spider):
data = {
"source_id": 'name_of_the_running_spider,
"token": "token",
…
m_sasha
- 239
- 1
- 7