4

Context:

I currently have a program which loads a set of plugins from their file paths (in a mapped network drive) using the method shown in another SO thread. These plugins are designed to be rolling release, which means I need constant access to writing them. The current mechanism locks the files so I have to ask everyone to close the software so I can update the files.

The question:

I was wondering if there was a way to, possibly using a similar method to that linked above, import a file from an io.BytesIO object of the plugin's raw contents (hence unlocking the file for me to make changes as I please).

More generally:

More specifically, can I keep the raw module contents in memory without touching a physical disk? If such a thing is not possible, is there a way to fully load these modules into memory so I can then unlock the files being imported?

As I have stated in my comment, I understand you can mount a virtual-filesystem on a Linux-based OS (which could have solved my problem), though sadly I developing for Windows and Microsoft can never make your life easy! :-)

Note:

I am not asking where I can copy these files to import them from a local version (e.g. temp, cache, etc.).

I understand this is quite a specialist question so any help is much appreciated

Minion Jim
  • 1,239
  • 13
  • 30
  • Interesting. Mostly out of curiosity, if you load the plugin (module?) in bytes, doesn’t this mean you’ll have to update the module’s code in bytes as well? (I’m sure I’m missing something here ...). – S3DEV Apr 13 '20 at 11:37
  • 1
    @S3DEV, I would read the modules from the network drive into a `BytesIO`, so while I am not reducing the R/W load on the drive, I am only keeping the files open for a minimal amount of time (with the number of users, I will statistically never have a problem) – Minion Jim Apr 13 '20 at 11:43
  • Am I understanding this correctly that you don't want to make local copies of the remote Python module files? Probably there is a compromise to be made by maintaining an _in memory_ copy of the files. Maybe with a combination of [`tempfile.SpooledTemporaryFile`](https://docs.python.org/3/library/tempfile.html#tempfile.SpooledTemporaryFile) and [`io.StringIO`](https://docs.python.org/3/library/io.html#io.StringIO). Or maybe with a 3rd party library, for example I randomly stumbled upon [_memory-tempfile_](https://pypi.org/project/memory-tempfile/). – sinoroc Apr 13 '20 at 12:37
  • @sinoroc, you are correct, I am trying to avoid copying the modules to the local disk and importing from an in-memory version would be ideal (or forcing Python to release the original file after import). I have no problem getting the modules into any of the listed formats but importing from these is where I cannot find a method (in the case of `tempfile.SpooledTemporaryFile`, I would have to write it the disk before it could be imported) – Minion Jim Apr 13 '20 at 16:01
  • If I could get a file-path that points to the object in memory without needing to actually be written, that would also solve the problem perfectly. I know on a Linux-based OS you can mount a file-system straight from the RAM, but I doubt Microsoft could be so helpful :-) – Minion Jim Apr 13 '20 at 16:10
  • I see. I guess, in your situation, I would try to follow the code paths and identify which part (if any) of `importlib` (or lower layers) really needs a file on the file system. Or identify why the files are not released after being imported in Python's module cache. Or maybe rethink the whole thing and accept that writing something on the local file system could be an acceptable compromise. – sinoroc Apr 13 '20 at 17:40
  • Thank you for your response. I clearly didn't do my initial research properly as I have found a thread discussing a similar thing (https://stackoverflow.com/a/14192708/7496549)! I may look to port the answer to Python3, + a few modification to be more applicable to this situation. If I get it working, I will post it here as an answer so someone else (could) benefit from it. I could also create a meta-class, though I would also rather avoid that! – Minion Jim Apr 13 '20 at 17:45

1 Answers1

1

While not being from an io.BytesIO object as I originally asked for, I was able to import a module from its source after finding this incredibly helpful article. I have not copied the code here as it is quite large, though I was able to get it to successfully import the virtual module.

The following code is after I modified the loader to remove the common prefix, and creates a class of the module by first executing the source, getting the globals from it and finally using Python's type method to create the module class.

It is not particularly pretty and definitely breaks some Python style recommendations, so I am definitely open to improvements!

source = """def hello():
    print("I don't want to say hi to the world")"""
name = "my_module"

glo = {}
exec(source, glo)

injector = DependencyInjector()
injector.provide(name, type(name, (), glo))
injector.install()

foo = __import__(name)
foo.hello()
Minion Jim
  • 1,239
  • 13
  • 30