Scrapy: cannot load items to spider

Question

I cannot load scrapy items to scrapy spiders. Here is my project structure:

-rosen
  .log
  .scrapers
    ..scrapers
      ...spiders
        ....__init__.py
        ....exampleSpider.py
      ...__init__.py
      ...items.py
      ...middlewares.py
      ...pipelines.py
      ...settings.py
  .src
    ..__init__.py
    ..otherStuff.py
  .tmp

This structure has been created using scrapy startproject scrapers inside of rosen project (directory).

Now, the items.py has the following code:

import scrapy
from Decimal import Decimal

class someItem(scrapy.Item):
   title: str = scrapy.Field(serializer=str)
   bid: Decimal = scrapy.Field(serializer=Decimal)

And the exampleSpider.py has the following code:

import scrapy
from __future__ import absolute_import

from scrapy.loader import ItemLoader
from scrapers.scrapers.items import someItem

class someSpider(scrapy.Spider):
   name = "some"

   def __init__(self, **kwargs):
       super().__init__(**kwargs)
       self._some_fields = someItem()

   def parse(self, response) -> None: 
       some_loader = ItemLoader(item=self._some_fields, response=response)
       print(self._some_fields.keys())

The error I get is the following: runspider: error: Unable to load 'someSpider.py': No module named 'scrapers.scrapers'

I found Scrapy: ImportError: No module named items and tried all three solutions by renaming and adding from __future__ import absolute_import. Nothing helps. Please advice.

The command that I execute is scrapy runspider exampleSpider.py. I tried it from the spiders and rosen directories.

Try `from ..items import someItem` instead of `from scrapers.scrapers.items import someItem` — thirdDeveloper, Apr 11 '20 at 18:58
Thanks! I have tried this as well and I get another error `runspider: error: Unable to load 'exampleSpider.py': attempted relative import with no known parent package` ` — abe, Apr 11 '20 at 19:07
@alienbeeq you need to run command `scrapy crawl some` that might work https://docs.scrapy.org/en/latest/topics/commands.html#std:command-crawl — Umair Ayub, Apr 12 '20 at 05:32
Well, this is another method to run a code as per documentation. You can either `scrapy runspider` (in this option no need to create a scrapy project) or `scrapy crawl` specifying the name of the crawler. I get the same error `ModuleNotFoundError: No module named 'scrapers.scrapers'` — abe, Apr 12 '20 at 08:25

score 0 · Answer 1 · answered Aug 17 '20 at 10:50

i do not see any virtualenv inside your directory. So i recommend you to do so eg. under 'rosen'. you can try this:

try:
    from scrapers.items import someItem
except FileNotFoundError:
    from scrapers. scrapers.items import someItem

then cal it with:

scrapy crawl NameOfSpider

or:

scrapy runspider path/to/spider.py

Scrapy: cannot load items to spider

1 Answers1