I try to use as much as possible functional programming, because this is more convenient and above all easier to read and to correct. Well, it was to explain the purpose of my source code.
tree diagram
scrapy_project
|_ scrapy.cfg
|_ scrapy_project
|_ __init__.py
|_ module_parsing.py
|_ module_correction.py
|_ items.py
|_ ...
|_ spiders
|_ crawling_spider.py
module_parsing.py
and module_correction.py
are customized modules of my own.
module_parsing.py and module_correction.py
In module_parsing.py
I have these kind of function.
import re
import datetime as dt
import dateparser
def noms_entraineurs(response, arg_disc):
"""Return a list of names"""
ent = parse instructions
return ent
and so on.
And because I am not perfect, I do some parsing errors and want to correct it, this is why for a specific method of my spider I think about creating a function with a dictionary inside it with the field as key and the function (parsing function) as value.
It gives something like this.
from .module_parsing import *
def fonction_parse_correction(
field, response, arg_desc, arg_disc, arg_true_false):
dico_fonction_parse = {
'discipline': discipline(arg_desc),
...
'entraineurs' : noms_entraineurs(response,
arg_disc),
...
}
return dico_fonction_parse[field]
Like this, I hope, I just have to write:
fonction_parse_correction(field, response, arg_desc, arg_disc, arg_true_false)
This what I tried in scrapy shell but it failed with NameError: name 'function_name' is not defined
trying in scrapy shell
I launched in the shell scrapy shell https://website.com
After I imported the customized modules as shown by Cameron it gives the following.
In [1]: import sys
In [2]: sys.path.insert(0,'/home/user/directory_a/.../scrapy_project/scrapy_project')
In [3]: from module_parsing import *
#and it worked well here
...
In [7]: from module_correction import *
...
In [10]: fonction_parse_correction('entraineurs',response,arg_desc,'attele',False)
---------------------------------------------------------------------------
NameError Traceback (most recent call last)
<ipython-input-10-137e645b094c> in <module>
----> 1 fonction_parse_correction('entraineurs',response,arg_desc,'attele',False)
~/.../scrapy_project/scrapy_project/module_correction.py in fonction_parse_correction(field, response, arg_desc, arg_disc, arg_true_false)
50
51 dico_fonction_parse = {
---> 52 'allocation': allocation_devise(response)[0],
53 'devise': allocation_devise(response)[1],
54 'hippodrome':hippodrome_numreu_prix(response)[0],
NameError: name 'allocation_devise' is not defined
So, I would like to know why I have got this NameError
while I explicitly imported module_parsing.py
in module_correction.py
to obtain all the functions and their names.
Notes: Ubuntu 18.04, Scrapy 1.5.2, Python 3.7.1