Reuse python loaded module

Question

Is it possible to reuse python module already loaded in memory ?

Let us say I have scripts loader.py and consume.py. I am try to do next thing - invoke loader.py and reuse it in consume.py . First script should load in memory big file, second one will be invoked many times and use big file.

Can I achieve this? I am not familar with python but I guess, there should be a way to access loaded module (script) in memory.

My current implementation attempt looks like this:

loader.py

   x = 3
   print 'module loaded'

consume.py

  from loader import x 
  print x

Update

I have tried to use importlib as it was described here and here, but my loader module loads every time again. Below is my code for cosume.py

import importlib

module = importlib.import_module('loader')

globals().update(
    {n: getattr(module, n) for n in module.__all__} if hasattr(module, '__all__')
    else
    {k: v for (k, v) in module.__dict__.items() if not k.startswith('_')
})

print(x)

Final goal

Invoke consume script many times from nodejs and not to load big file every time. Need to share data between script executions

Possible duplicate of [Dynamic module import in Python](https://stackoverflow.com/questions/301134/dynamic-module-import-in-python) — sam-pyt, Oct 29 '17 at 21:41

Josh Karpel · Answer 1 · 2017-10-29T22:11:29.540

Define a function in consume.py that does the work you want to do. In fact, it should all be functions. You could have three files, one where you define functions that load the data, one where you define functions that consume data, and one where you combine them into some process.

For example, one module loads data:

# loader.py

def load_data():
    # load the data

One module where you write functions that consume data:

# consume.py

def consume_data(data):
    # do stuff with the data

def consume_data_differently(data):
    # do other stuff with the data

and a script that actually does stuff:

# do_stuff.py

from loader import load_data
from consume import consume_data

data = load_data()

for d in data:  # consume pieces of data in a loop
    consume_data(d)

Settings things up like this gives you much more flexibility than relying on the import mechanism to run code, which isn't what it's designed for.

Addendum based on your update: you're making things much harder than they need to be. You really, really don't need to play around with importlib and globals() in normal code. Those are tools for building libraries, not doing data analysis.

There is no place for do_stuff.py file . I am planning to invoke consume.py file from node.js many times. And at this time all settings (python objects for reuse) should be already loaded. I am misunderstand how to integrate proposed solution. Thanks. — Anton Putau, Oct 29 '17 at 22:10
What do you mean "no space"? Presumably if you can invoke `consume.py` many times, you could invoke `do_stuff.py` once and have it stream output to something? Could you add more about the program flow you're envisioning to the question? — Josh Karpel, Oct 29 '17 at 22:13

Reuse python loaded module

1 Answers1