Advice on Structuring ETL Application Code

Question

I need some advice on the best practice for structuring some code for an ETL application.

My structure currently is something along the lines of

main_process.py

mid_level_process.py

setup.py

   __init__.py

   module1.py

   modules2.py

   ...

My issue is these 'mid level processes' call a number of the functions packaged up in the lower level of the dir structure, however for maintainability & readability, it makes more sense to call them as functions themselves in the main process. Is it bad practice to also house these in the lower level too? It feels like bad practice to have these modules calling functions from each other? Or would it be better to define the functions in the main_process.py & then just call them there?

bad_coder · Accepted Answer · 2020-09-02T00:09:57.130

There's no hard rule here except avoiding circular imports.

It's what the __init__.pt exposes that makes for a package, and that can be largely independent from how you structure your directories. (But let's ignore the __init__.py and assume it is exists and is empty, thus you have a package with the exact layout of the file system.)

however for maintainability & readability, it makes more sense to call them as functions themselves in the main process.

No problem. Main_process.py can import from mid_level_process.py and from module1.py or module2.py nothing circular there so far.

Is it bad practice to also house these in the lower level too?

If both Main_process.py and mid_level_process.py are importing from module1.py and module2.py that makes sense. You can do that and have them in a separate package. (Nothing against one module importing another while both use for example int, right?!)

If you choose to call the latter two with mid_level_process.py as an intermediary that's valid, but if you also want to import them in Main_process.py that's also valid.

The issue of putting module1 and module2 on the same level with the others 2 modules is subjective to an extent and depends on context. Is there some conceptual modularity or cause for keeping them separate? Would some other module in a completely different context want to import module1 and module2 with no need for main or intermediary? Then there's no compelling need to have them in the same package or on the same directory level.

You current arrangement is what makes sense.

Advice on Structuring ETL Application Code

1 Answers1