1

I try to use as much as possible functional programming, because this is more convenient and above all easier to read and to correct. Well, it was to explain the purpose of my source code.

tree diagram

scrapy_project
 |_ scrapy.cfg
 |_ scrapy_project
    |_ __init__.py
    |_ module_parsing.py
    |_ module_correction.py
    |_ items.py
    |_ ...
    |_ spiders
        |_ crawling_spider.py

module_parsing.py and module_correction.py are customized modules of my own.

module_parsing.py and module_correction.py

In module_parsing.py I have these kind of function.

import re
import datetime as dt
import dateparser

def noms_entraineurs(response, arg_disc):
    """Return a list of names"""
    ent = parse instructions
    return ent

and so on.

And because I am not perfect, I do some parsing errors and want to correct it, this is why for a specific method of my spider I think about creating a function with a dictionary inside it with the field as key and the function (parsing function) as value.

It gives something like this.

from .module_parsing import *
def fonction_parse_correction(
    field, response, arg_desc, arg_disc, arg_true_false):

    dico_fonction_parse = {
    'discipline': discipline(arg_desc),
    ...
    'entraineurs' : noms_entraineurs(response,
                                arg_disc),
    ...
    }
    return dico_fonction_parse[field]

Like this, I hope, I just have to write:

fonction_parse_correction(field, response, arg_desc, arg_disc, arg_true_false)

This what I tried in scrapy shell but it failed with NameError: name 'function_name' is not defined

trying in scrapy shell

I launched in the shell scrapy shell https://website.com

After I imported the customized modules as shown by Cameron it gives the following.

In [1]: import sys                                                              

In [2]: sys.path.insert(0,'/home/user/directory_a/.../scrapy_project/scrapy_project')
In [3]: from module_parsing import * 
#and it worked well here
...
In [7]: from module_correction import * 
...
In [10]: fonction_parse_correction('entraineurs',response,arg_desc,'attele',False)

---------------------------------------------------------------------------
NameError                                 Traceback (most recent call last)
<ipython-input-10-137e645b094c> in <module>
----> 1 fonction_parse_correction('entraineurs',response,arg_desc,'attele',False)

~/.../scrapy_project/scrapy_project/module_correction.py in fonction_parse_correction(field, response, arg_desc, arg_disc, arg_true_false)
     50 
     51     dico_fonction_parse = {
---> 52     'allocation': allocation_devise(response)[0],
     53     'devise': allocation_devise(response)[1],
     54     'hippodrome':hippodrome_numreu_prix(response)[0],

NameError: name 'allocation_devise' is not defined

So, I would like to know why I have got this NameError while I explicitly imported module_parsing.py in module_correction.py to obtain all the functions and their names.

Notes: Ubuntu 18.04, Scrapy 1.5.2, Python 3.7.1

marc_s
  • 732,580
  • 175
  • 1,330
  • 1,459
AvyWam
  • 890
  • 8
  • 28

0 Answers0