9

In order to reduce development time of my Python based web application, I am trying to use reload() for the modules I have recently modified. The reload() happens through a dedicated web page (part of the development version of the web app) which lists the modules which have been recently modified (and the modified time stamp of py file is later than the corresponding pyc file). The full list of modules is obtained from sys.modules (and I filter the list to focus on only those modules which are part of my package).

Reloading individual python files seems to work in some cases and not in other cases. I guess, all the modules which depend on a modified module should be reloaded and the reloading should happen in proper order.

I am looking for a way to get the list of modules imported by a specific module. Is there any way to do this kind of introspection in Python?

I understand that my approach might not be 100% guaranteed and the safest way would be to reload everything, but if a fast approach works for most cases, it would be good enough for development purposes.

Response to comments regarding DJango autoreloader

@Glenn Maynard, Thanx, I had read about DJango's autoreloader. My web app is based on Zope 3 and with the amount of packages and a lot of ZCML based initializations, the total restart takes about 10 seconds to 30 seconds or more if the database size is bigger. I am attempting to cut down on this amount of time spent during restart. When I feel I have done a lot of changes, I usually prefer to do full restart, but more often I am changing couple of lines here and there for which I do not wish to spend so much of time. The development setup is completely independent of production setup and usually if something is wrong in reload, it becomes obvious since the application pages start showing illogical information or throwing exceptions. Am very much interested in exploring whether selective reload would work or not.

Shailesh Kumar
  • 6,457
  • 8
  • 35
  • 60
  • It's much safer to do something like Django's autoreloader, which re-executes the backend entirely when a source file is modified. I don't know of any disadvantage; you modify a file and everything's reloaded a second or two later, automatically. Something which only works in "most" cases is very bad for development; you're just asking to be bitten painfully down the road when it doesn't. – Glenn Maynard Dec 01 '09 at 22:21
  • Revisiting here due to a duplicate question, and adding that "the way to do minimize reload time when using zope" is to use sauna.reload by now (2013) – jsbueno Mar 19 '13 at 18:03

3 Answers3

5

So - this answers "Find a list of modules which depend on a given one" - instead of how the question was initally phrased - which I answered above.

As it turns out, this is a bit more complex: One have to find the dependency tree for all loaded modules, and invert it for each module, while preserving a loading order that would not break things.

I had also posted this to brazillian's python wiki at: http://www.python.org.br/wiki/RecarregarModulos

#! /usr/bin/env python
# coding: utf-8

# Author: João S. O. Bueno
# Copyright (c) 2009 - Fundação CPqD
# License: LGPL V3.0


from types import ModuleType, FunctionType, ClassType
import sys

def find_dependent_modules():
    """gets a one level inversed module dependence tree"""
    tree = {}
    for module in sys.modules.values():
        if module is None:
            continue
        tree[module] = set()
        for attr_name in dir(module):
            attr = getattr(module, attr_name)
            if isinstance(attr, ModuleType):
                tree[module].add(attr)
            elif type(attr) in (FunctionType, ClassType):        
                tree[module].add(attr.__module__)
    return tree


def get_reversed_first_level_tree(tree):
    """Creates a one level deep straight dependence tree"""
    new_tree = {}
    for module, dependencies in tree.items():
        for dep_module in dependencies:
            if dep_module is module:
                continue
            if not dep_module in new_tree:
                new_tree[dep_module] = set([module])
            else:
                new_tree[dep_module].add(module)
    return new_tree

def find_dependants_recurse(key, rev_tree, previous=None):
    """Given a one-level dependance tree dictionary,
       recursively builds a non-repeating list of all dependant
       modules
    """
    if previous is None:
        previous = set()
    if not key in rev_tree:
        return []
    this_level_dependants = set(rev_tree[key])
    next_level_dependants = set()
    for dependant in this_level_dependants:
        if dependant in previous:
            continue
        tmp_previous = previous.copy()
        tmp_previous.add(dependant)
        next_level_dependants.update(
             find_dependants_recurse(dependant, rev_tree,
                                     previous=tmp_previous,
                                    ))
    # ensures reloading order on the final list
    # by postponing the reload of modules in this level
    # that also appear later on the tree
    dependants = (list(this_level_dependants.difference(
                        next_level_dependants)) +
                  list(next_level_dependants))
    return dependants

def get_reversed_tree():
    """
        Yields a dictionary mapping all loaded modules to
        lists of the tree of modules that depend on it, in an order
        that can be used fore reloading
    """
    tree = find_dependent_modules()
    rev_tree = get_reversed_first_level_tree(tree)
    compl_tree = {}
    for module, dependant_modules in rev_tree.items():
        compl_tree[module] = find_dependants_recurse(module, rev_tree)
    return compl_tree

def reload_dependences(module):
    """
        reloads given module and all modules that
        depend on it, directly and otherwise.
    """
    tree = get_reversed_tree()
    reload(module)
    for dependant in tree[module]:
        reload(dependant)

This wokred nicely in all tests I made here - but I would not recoment abusing it. But for updating a running zope2 server after editing a few lines of code, I think I would use this myself.

jsbueno
  • 99,910
  • 10
  • 151
  • 209
  • Hi, I figured out that just looking for ModuleType attributes in the dir(module) is not good enough. A number of times the imports look like ``from xyz import abc``. To handle this, one should also consider FunctionType and ClassType attributes in the dir(module) listing and for those ones, one should pickup their corresponding getattr(attr, '__module__') and add them in dependencies – Shailesh Kumar Dec 17 '09 at 14:58
  • Indeed. I will have to fix that - or remove the code from both places - it is so compelx now it "have" to work for whoever needs it. – jsbueno Dec 18 '09 at 12:09
  • ClassType is no longer a type in 3.4 (probably sooner), safe to assume MethodType would be it's replacement? – Brian S Oct 04 '16 at 17:10
3

Some introspection to the rescue:

from types import ModuleType

def find_modules(module, all_mods = None):
   if all_mods is None:
      all_mods = set([module])
   for item_name in dir(module):
       item = getattr(module, item_name)
       if isinstance(item, ModuleType) and not item in all_mods:
           all_mods.add(item)
           find_modules(item, all_mods)
   return all_mods

This gives you a set with all loaded modules - just call the function with your first module as a sole parameter. You can then iterate over the resulting set reloading it, as simply as: [reload (m) for m in find_modules(<module>)]

jsbueno
  • 99,910
  • 10
  • 151
  • 209
  • Just trying to understand this code. - start with a given module x - create an empty set of modules which have been imported by x - iterated over dir(x) to identify all items which happen to be modules - add them to the set of modules for x - do this recursively to find all dependencies for x I would probably need to start with this and get a reverse mapping done to identify set of all modules which depend on a specific module – Shailesh Kumar Dec 02 '09 at 07:09
  • Wait - -do you need a "list of modules a module depends upon in python" or a list "of all modules which depend on a specific module"? I can come up with code for the later, not much more complex than this one - but the question is phrased for the former. – jsbueno Dec 02 '09 at 10:51
  • @jsbueno Yes u r right, I phrased the subject line incorrectly :( I have revised the subject line now. I am looking for all modules whichc depend on a specific module. – Shailesh Kumar Dec 02 '09 at 15:57
2

You might want to take a look at Ian Bicking's Paste reloader module, which does what you want already:

http://pythonpaste.org/modules/reloader?highlight=reloader

It doesn't give you specifically a list of dependent files (which is only technically possible if the packager has been diligent and properly specified dependencies), but looking at the code will give you an accurate list of modified files for restarting the process.

Douglas Mayle
  • 21,063
  • 9
  • 42
  • 57