4

I need to import python modules by filepath (e.g., "/some/path/to/module.py") known only at runtime and ignore any .pyc files should they exist.

This previous question suggests the use of imp.load_module as a solution but this approach also appears to use a .pyc version if it exists.

importme.py

SOME_SETTING = 4

main.py:

import imp
if __name__ == '__main__':
    name = 'importme'
    openfile, pathname, description = imp.find_module(name)
    module = imp.load_module(name, openfile, pathname, description)
    openfile.close()
    print module

Executing twice, the .pyc file is used after first invocation:

$ python main.py 
<module 'importme' from '/Users/dbryant/temp/pyc/importme.py'>

$ python main.py 
<module 'importme' from '/Users/dbryant/temp/pyc/importme.pyc'>

Unfortunately, imp.load_source has the same behavior (from the docs):

Note that if a properly matching byte-compiled file (with suffix .pyc or .pyo) exists, it will be used instead of parsing the given source file.

Making every script-containing directory read-only is the only solution that I know of (prevents generation of .pyc files in the first place) but I would rather avoid it if possible.

(note: using python 2.7)

Community
  • 1
  • 1
awesomo
  • 8,526
  • 2
  • 21
  • 24
  • 1
    Why do you want to prevent use of pyc files? It will only use the pyc file if it's the same as the py. If you change the py, it will recompile the pyc. – Falmarri Dec 07 '11 at 20:12
  • @Falmarri It's complicated. It has to do with the fact that different types of machines are accessing the same set of python files and they seem to produce incompatible byte-compilations of the same source code (such that the pyc generated by machine A is not readable by machine B, causing machine B to fail to import the module dynamically); and yes, all machines are using the interpreter binary. – awesomo Dec 07 '11 at 20:15
  • The only reason different machines would write different bytecode is if they're running different versions of the interpreter. – limscoder Dec 07 '11 at 21:18

3 Answers3

2

load_source does the right thing for me, i.e. 

dir, name = os.path.split(path)
mod = imp.load_source(name, path)

uses the .py variant even if a pyc file is available - name ends in .py under python3. The obvious solution is obviously to delete all .pyc files before loading the file - the race condition may be a problem though if you run more than one instance of your program.

One other possibility: Iirc you can let python interpret files from memory - i.e. load the file with the normal file API and then compile the in-memory variant. Something like:

path = "some Filepath.py"
with open(path, "r", encoding="utf-8") as file:
    data = file.read()
exec(compile(data, "<string>", "exec")) # fair use of exec, that's a first!
Voo
  • 29,040
  • 11
  • 82
  • 156
  • Nice, I didn't even consider exec'ing the settings file/module contents. This would also be a workable solution for me. – awesomo Dec 07 '11 at 21:58
0

How about using zip files containing python sources instead: 

import sys
sys.path.insert("package.zip")
ILYA Khlopotov
  • 705
  • 3
  • 5
  • Interesting, but doesn't really work well for my use-case. The modules that I'm dynamically loading are settings files and need to be easily human-readable and editable. – awesomo Dec 07 '11 at 20:13
  • 2
    You could also look at using PYTHONDONTWRITEBYTECODE environment variable. – ILYA Khlopotov Dec 07 '11 at 20:29
  • Good point. That would also work, but the only pyc files that are causing issues are those that are dynamically loaded with `imp` and I would like to be able to still use byte-code versions for everything else when available. – awesomo Dec 07 '11 at 20:34
  • 3
    Could set that environment variable using sys.dont_write_bytecode, imp your modules and then change the bytecode setting back to the way it was to achieve this? – Sanjamal Dec 07 '11 at 21:23
  • @Sanjamal Thanks, if your suggestion works then yes it would be sufficient for me. I will test it. – awesomo Dec 07 '11 at 21:31
  • @Sanjamal Your solution works perfectly. If you submit it as an answer I'll gladly accept it. – awesomo Dec 07 '11 at 22:05
0

You could mark the directories containing the .py files as read-only.

limscoder
  • 3,037
  • 2
  • 23
  • 37
  • Yes, I know (see bottom of question) but this isn't really solving the problem and is very inconvenient because I do need to still write to the containing directories for logs, outputs, etc. – awesomo Dec 07 '11 at 20:21
  • I don't know what you're exact situation is, but I usually try to avoid writing data to the same directory that contains code. – limscoder Dec 07 '11 at 21:14
  • I agree in principle, but python settings files blur the line between data and code in this case (very common practice in Django for example) – awesomo Dec 07 '11 at 21:29