Failing at importing modules on items.py file - Scrapy

Question

I'm new in Python and I'm currently learning webscraping with spiders. Following the tutorial, I stucked at relative importing with Python.

This is the structure of my current folder (provided by scrapy startproject p1):

My items.py file:

# Define here the models for your scraped items
#
# See documentation in:
# https://docs.scrapy.org/en/latest/topics/items.html

import scrapy


class Test(scrapy.Item):
    # define the fields for your item here like:
    # name = scrapy.Field()
    title = scrapy.Field()
    price = scrapy.Field()
    pass

In my filetwo.py, it contains:

import scrapy
from p1.items import Test

When I run the code, I get "ModuleNotFoundError: No module named 'p1'"

I also read some people online that faced the same problem, so I tried ..items import Test and still didn't work. It gave me the error: ImportError: attempted relative import with no known parent package

Can someone give me a light?

Thanks in advance!

Refer here: https://stackoverflow.com/questions/16780014/import-file-from-parent-directory — Maas, Jul 08 '21 at 00:18

Mr_and_Mrs_D · Answer 1 · 2021-07-09T05:34:56.733

1

The answer you accepted is wrong unfortunately. You should not be messing with the sys.path. You should instead:

$ cd P1
$ python -m p1.spiders.filetwo # note no .py

You are obviously running the filetwo.py script directly from inside the spiders package - that's an antipattern in python leading to all these errors. Messing with the sys.path on the other hand can lead to a slew of subtle bugs.

edited Jul 09 '21 at 05:34

answered Jul 08 '21 at 10:17

Mr_and_Mrs_D

32,208
39
178
361

First, thanks for the answer! But, I'm not sure if I understood it fully. The problem I am facing (or that I think i'm facing) is with relative importing. Your instructions were about running the program, right? How should I relative import my class "Test" in items.py file? – Daniel Siman Jul 16 '21 at 00:59
@DanielSiman how you run the program (or rather the module) affects how the imports are resolved. So relative or absolute imports are made to work when running the module they are in, as I describe here (pycharm run configurations support the -m switch btw). Both `from p1.items import Test` and `from ..items import Test` should work – Mr_and_Mrs_D Jul 19 '21 at 12:05

score 0 · Accepted Answer · answered Jul 08 '21 at 00:21

0

I am sure my comment should help you. But here is an example

answered Jul 08 '21 at 00:21

Maas

1,317
9
20

Failing at importing modules on items.py file - Scrapy

2 Answers2