Python -- reloading module level objects

Question

What I want to do

I'm trying to write unit test to check that my regexp is constructed properly:

# mediamanager/models.py

import re

from django.conf import settings

filetypes_re = {}
for key, exts in settings.MM_FILETYPES.items():
    filetypes_re[key] = re.compile(r'({})'.format('|'.join(exts)))

NOTE: Actually, I'm not sure why I'm writing this unittest at all, since this piece of code is more than straighforward… But that's not the point.

As you can see, the final regexp depends on variable settings.MM_FILETYPES which can be set by user. I need to test specific input and for this case, Django provides decorator @override_settings which temporary overrides settings values:

# mediamanager/tests.py

import unittest
from django.test.utils import override_settings
from mediamanager.models import (filetypes_re, …)  # import everything we want to test

class ModelsTestCase(unittest.TestCase):
    @override_settings(MM_FILETYPES={'image': ['jpg', 'png', 'gif'],
                                     'document': ['pdf', 'txt'],
                                     'audio': ['mp3', 'wav']})
    def test_filetype_re(self):
        filetypes_re_exp = {'image': '(jpg|png|gif)',
                            'document': '(pdf|txt)',
                            'audio': '(mp3|wav)'}

        for key, value in filetypes_re_exp.items():
            self.assertEqual(value, filetypes_re[key].pattern)

This test unfortunatelly don't pass. Module mediamanager.models is loaded before the settings are overriden and therefore the filetypes_re is compiled using old settings. I need to reload it (somehow) to new settings take effect.

Problem

I altered unittest in this way:

@override_settings(MM_FILETYPES={'image': ['jpg', 'png', 'gif'],
                                 'document': ['pdf', 'txt'],
                                 'audio': ['mp3', 'wav']})
def test_filetype_re(self):
    import mediamanager.models  # obtaining module object from sys.modules have the same result
    reload(mediamanager.models)
    filetypes_re = mediamanager.models.filetypes_re
    filetypes_re_exp = {'image': '(jpg|png|gif)',
                        'document': '(pdf|txt)',
                        'audio': '(mp3|wav)'}

    for key, value in filetypes_re_exp.items():
        self.assertEqual(value, filetypes_re[key].pattern)

And test passed. But probably because I imported other objects from mediamanager.models module, other tests in this test case failed. Not all of them, only two (which is strange). EDIT: It's not strange at all. Only fails tests which runs after test_filetyes_re and reload() call.

Questions

How to 'reload' module mediamanager.models in a way that:

filetypes_re are evaluated with new settings?
All other objects imported from same model remains unaffected?

Should I rewrite a piece of code only because it became unstable after reload (I mean other those objects from mediamanager.models which don't pass test after reload)? I've read some articles about reloading modules that usually it's not a good idea.

Is there a better way to define module level objects, like this regexp, to make testing easier?

Can you explain a little more about why you need to reload the module at all? Why did you change the first version of your test? — scytale, Jul 23 '12 at 19:08
That's because user define MM_FILETYPES on his own. Therefore I don't know what should I test. Temporary I override these settings (using decorator), but since the mediamanager.models is already loaded, the filetypes_re is evaluated using old settings. I need to reload him to new settings take effect. — Tomáš Ehrlich, Jul 23 '12 at 19:12
ah ok - you need to clear some things up - the first block of code in your question - is that from your `settings.py`? or somewhere else? Secondly all your unit test is doing is checking whether `mediamanager.models.filetypes_re` matches what you expect. Why? Is mediamanager your app? — scytale, Jul 23 '12 at 19:20
No, settings.py are irelevant. Tests should pass for every settings, that's why I'm using override_settings decorator -- to create known inputs. In this unittest, only this method (test_filetypes_re) checks filetypes_re. Other methods checks other objects from mediamanager.models and that's the reason, why those tests failed. After module reload, all objects imported from that module, are still pointing to old ones. — Tomáš Ehrlich, Jul 23 '12 at 19:26
oh i get it - you want `mediamanager.models.filetypes_re` to be generated using a the overridden version of `settings.MM_FILETYPES` - you really should mention that explicitly - there is a lot of verbiage in your question and it's hard to see what the key issue is. — scytale, Jul 23 '12 at 19:34
No, it hasn't. It only reloads. I've read in one article about module reload, that all objects loaded using 'from module import …' are the same (old), even after module reload. Thus, I thougt that's the problem, but probably not. Just my assumption, sorry. — Tomáš Ehrlich, Jul 23 '12 at 19:35
Oh, sorry. I added new paragraph right after you've asked first question: This test unfortunatelly don't pass. Module mediamanager.models is loaded before the settings are overriden and therefore the filetypes_re is compiled using old settings. I need to reload it (somehow) to new settings take effect. — Tomáš Ehrlich, Jul 23 '12 at 19:36

score 0 · Accepted Answer · answered Jul 23 '12 at 19:40

0

there are a couple of ways. off the top of my head:

# mediamanager/models.py

import re

from django.conf import settings

def get_filetypes_re(mm_filetypes=settings.MM_FILETYPES):
    filetypes_re = {}
    for key, exts in settings.MM_FILETYPES.items():
    filetypes_re[key] = re.compile(r'({})'.format('|'.join(exts)))
    return filetypes_re

and your test:

MM_FILETYPES={'image': ['jpg', 'png', 'gif'],
                        'document': ['pdf', 'txt'],
                        'audio': ['mp3', 'wav']})

def test_filetype_re(self):
    filetypes_re = mediamanager.models.get_filetypes_re(mm_filetypes=MM_FILETYPES)
    filetypes_re_exp = {'image': '(jpg|png|gif)',
                        'document': '(pdf|txt)',
                        'audio': '(mp3|wav)'}

    for key, value in filetypes_re_exp.items():
        self.assertEqual(value, filetypes_re[key].pattern)

answered Jul 23 '12 at 19:40

scytale

12,346
3
32
46

That crossed my mind too, but: I'm compiling this regexp so I don't have to compile it everytime again and save a little bit of time. There are more regexps on top of my module. – Tomáš Ehrlich Jul 23 '12 at 19:43
I really doubt you would be able to notice the microseconds you would save. But if you really want to micro-optimise you could just memoize the function's return value. – scytale Jul 23 '12 at 19:44
1

oh and FYI python caches the most recent few hundred compiled regexes - I doubt even the half dozen lines needed to implement memoization would make a mesurable difference in execution time. http://stackoverflow.com/a/452143/473285 – scytale Jul 23 '12 at 19:53
In that case, you answer is acceptable for me. I was just thinking about defining filetypes_re = get_filetypes_re() on the top of the module, but if python caches regexps internally, there's no need for that… Thanks! – Tomáš Ehrlich Jul 23 '12 at 20:55
this would be better than defining a module level variable: http://stackoverflow.com/a/1988826/473285 – scytale Jul 23 '12 at 21:04
and you're totally thinking about optimization too soon. I'm willing to bet that even without the caching the regex compilation time would not be significant. Your app probably spends an order of magnitude more time on database and network operations. Write good code first. Only optimize once you *know* (by profiling) that you need to. – scytale Jul 23 '12 at 21:08
This is not about optimalization. It's just my attitude -- when I see method called 'compile' I want to run it as less as possible. Using memoize decorator doesn't seem appropriate here, because, as you mentioned, compiled regexps are cached internally. My question was more about the problem with reload, than about optimalization. I still don't understand why my code doesn't work after reload and that's bigger problem. – Tomáš Ehrlich Jul 23 '12 at 22:16
let us [continue this discussion in chat](http://chat.stackoverflow.com/rooms/14311/discussion-between-scytale-and-elvard) – scytale Jul 23 '12 at 23:10

Python -- reloading module level objects

What I want to do

Problem

Questions

1 Answers1