Detect if a python module changes and then reload

Question

So I'm developing a rather large python project with a number of modules. The "main" (runnable) module is a daemon (a Thrift daemon, actually) which calls off to other modules for its actual functionality. Starting up the daemon takes a long time because some of the modules have a rather lengthy and involved initialization processes.

So when I start the daemon, I wait... let's say... 2 minutes for everything to load, which isn't too bad in the grand scheme of things. However for development it becomes a major pain because I need to restart the daemon EVERY TIME which has been wasting a lot of my time.

Most modules only takes a few seconds to load. Ideally what I'd like to do is detect when any of the files in a particular module have changed, and reload that particular module. I've already figured out how to reload a module, but at this point I can't figure out how to watch a particular module for changes. Keep in mind that a module isn't a single .py file in this case, but rather a directory with __init__.py and 5-10 .py files, so I need to detect when any of them have changed.

Here is the project layout (if it makes any difference at all)

project
| -- daemonize.py
| -- main.py
| -- moduleA
|    | -- __init__.py
|    | -- happy_panda.py
|    ` -- sad_panda.py
| -- moduleB
|    | -- __init__.py
|    | -- takes_forever_to_load.py
|    ` -- seriously_get_some_coffee.py
| -- moduleC
|    | -- __init__.py
|    | -- frequently_changes.py
|    | -- reasons_i_hate_my_job.txt
|    ` -- home_address_of_moduleB_developer.txt
` -- service.py <-- uses modules A, B, and C

Any ideas or suggestions are appreciated.

EDIT

Thanks for the great feedback. Here is the code I created based on the suggestions. There's a small bug where pyinotify seems to be getting more than one notification, but it's a very small problem for me so I'm not going to fix it.

https://gist.github.com/1013122

Doesn't reload already check that the file has changed before doing anything? — Bastien Léonard, Jun 07 '11 at 19:19
@Bastien: After looking at the documentation for reload, no I don't believe that's how reload behaves. Correct me if I'm wrong. — Chris Eberle, Jun 07 '11 at 19:23

score 9 · Accepted Answer · edited May 23 '17 at 12:00

9

Detect File Change Without Polling

Coupled with you already knowing how to reload your module this answer pretty much fills it out. It uses Inotify to "notify" (see what they did there) the program when the file is modified.

edited May 23 '17 at 12:00

Community

1
1

answered Jun 07 '11 at 19:23

Paystey

3,287
2
18
32

Very nice! I was thinking of trying to figure out if a module changed right before I call it, however doing it this way I can just reload the moment a file changes. Wouldn't have thought of that. – Chris Eberle Jun 07 '11 at 19:27
Glad to help, and glad I found your question actually, this'll come in handy down the line for something for I'm working on and I hadn't considered this technique as a time saver before. – Paystey Jun 07 '11 at 21:26

score 1 · Answer 2 · answered Apr 08 '20 at 16:10

1

It's an old question, but if you stumble upon it, here is a solution if you use IPython:

Enter the following lines in the IPython prompt at any time

%load_ext autoreload
%autoreload 2

For more options, check out this link: https://ipython.readthedocs.io/en/stable/

answered Apr 08 '20 at 16:10

PierreE

675
1
11
23

1

Here is the source code if you want to browse how they implemented it, and see some caveats: https://github.com/ipython/ipython/blob/master/IPython/extensions/autoreload.py – Nick Crews Mar 07 '22 at 15:49

score 1 · Answer 3 · edited May 23 '17 at 12:24

1

I would look at all of the files, and detect if a file was modified. If it was, I would reload it.

edited May 23 '17 at 12:24

Community

1
1

answered Jun 07 '11 at 19:19

Nick ODell

15,465
3
32
66

How would you handle determining which files belong to a module? From the top level all I have is the module name. – Chris Eberle Jun 07 '11 at 19:20
I would look into the folder recursively. Are there files that change often that aren't code or configuration files? – Nick ODell Jun 07 '11 at 19:22
Also, what are you doing on startup that takes 2 minutes? – Nick ODell Jun 07 '11 at 19:23
Aside from reasons_i_hate_my_job.txt, no not really – Chris Eberle Jun 07 '11 at 19:24
Oh and you don't want to know. Part of it is loading a number of very large objects into memory. – Chris Eberle Jun 07 '11 at 19:24

Detect if a python module changes and then reload

3 Answers3

Linked