Scrapy - new instance of Item Pipeline classes per process/job?

Question

I use Scrapyd for scheduling and launching spider jobs.

In Item Pipelines classes i set job specific variables into the class, which should not be shared by other spiders/jobs.

So my question is, does Scrapy/Scrapyd create new instance of pipeline class for each spider job/process?

score 1 · Answer 1 · answered Sep 25 '20 at 14:09

1

Scrapy/Scrapyd create new instance of pipelines, middlewares, etc for each job/process.

Hovewer your pipelines must not have static(or per class variables) on some conditions data can be accessed from other spider via python class variable.

answered Sep 25 '20 at 14:09

Georgiy

3,158
1
6
18

Under what conditions? – Mon B. Sep 25 '20 at 14:17
If your will store data inside pipeline class variable – Georgiy Sep 25 '20 at 14:21

Scrapy - new instance of Item Pipeline classes per process/job?

1 Answers1