We are building a reusable data framework using PySpark. As part of this, we had built one big utilities package that hosted all the methods. But now, we are planning to split it apart into smaller and more manageable packages.
How do we share the Spark Session and the Logger object across all the packages/modules?
