Constructing efficient functions in Python

Question

My programming is almost all self taught, so I apologise in advance if some of my terminology is off in this question. Also, I am going to use a simple example to help illustrate my question, but please note that the example itself is not important, its just a way to hopefully make my question clearer.

Imagine that I have some poorly formatted text with a lot of extra white space that I want to clean up. So I create a function that will replace any groups of white space characters that has a new line character in it with a single new line character and any other groups of white space characters with a single space. The function might look like this

def white_space_cleaner(text):
    new_line_finder = re.compile(r"\s*\n\s*")
    white_space_finder = re.compile(r"\s\s+")
    text = new_line_finder.sub("\n", text)
    text = white_space_finder.sub(" ", text)
    return text

That works just fine, the problem is that now every time I call the function it has to compile the regular expressions. To make it run faster I can rewrite it like this

new_line_finder = re.compile(r"\s*\n\s*")
white_space_finder = re.compile(r"\s\s+")
def white_space_cleaner(text):
    text = new_line_finder.sub("\n", text)
    text = white_space_finder.sub(" ", text)
    return text

Now the regular expressions are only compiled once and the function runs faster. Using timeit on both functions I find that the first function takes 27.3 µs per loop and the second takes 25.5 µs per loop. A small speed up, but one that could be significant if the function is called millions of time or has hundreds of patterns instead of 2. Of course, the downside of the second function is that it pollutes the global namespace and makes the code less readable. Is there some "Pythonic" way to include an object, like a compiled regular expression, in a function without having it be recompiled every time the function is called?

Answer 1

Keep a list of tuples (regular expressions and the replacement text) to apply; there doesn't seem to be a compelling need to name each one individually.

finders = [
    (re.compile(r"\s*\n\s*"), "\n"),
    (re.compile(r"\s\s+"), " ")
]
def white_space_cleaner(text):
    for finder, repl in finders:
        text = finder.sub(repl, text)
    return text

You might also incorporate functools.partial:

from functools import partial
replacers = {
    r"\s*\n\s*": "\n",
    r"\s\s+": " "
}
# Ugly boiler-plate, but the only thing you might need to modify
# is the dict above as your needs change.
replacers = [partial(re.compile(regex).sub, repl) for regex, repl in replacers.iteritems()]


def white_space_cleaner(text):
    for replacer in replacers:
        text = replacer(text)
    return text

Answer 2

Another way to do it is to group the common functionality in a class:

class ReUtils(object):
    new_line_finder = re.compile(r"\s*\n\s*")
    white_space_finder = re.compile(r"\s\s+")

    @classmethod
    def white_space_cleaner(cls, text):
        text = cls.new_line_finder.sub("\n", text)
        text = cls.white_space_finder.sub(" ", text)
        return text

if __name__ == '__main__':
   print ReUtils.white_space_cleaner("the text")

It's already grouped in a module, but depending on the rest of the code a class can also be suitable.

Answer 3

You could put the regular expression compilation into the function parameters, like this:

def white_space_finder(text, new_line_finder=re.compile(r"\s*\n\s*"),
                             white_space_finder=re.compile(r"\s\s+")):
    text = new_line_finder.sub("\n", text)
    text = white_space_finder.sub(" ", text)
    return text

Since default function arguments are evaluated when the function is parsed, they'll only be loaded once and they won't be in the module namespace. They also give you the flexibility to replace those from calling code if you really need to. The downside is that some people might consider it to be polluting the function signature.

I wanted to try timing this but I couldn't figure out how to use timeit properly. You should see similar results to the global version.

Markus's comment on your post is correct, though; sometimes it's fine to put variables at module-level. If you don't want them to be easily visible to other modules, though, consider prepending the names with an underscore; this marks them as module-private and if you do from module import * it won't import names starting with an underscore (you can still get them if you ask from them by name, though).

Always remember; the end-all to "what's the best way to do this in Python" is almost always "what makes the code most readable?" Python was created, first and foremost, to be easy to read, so do what you think is the most readable thing.

Answer 4

In this particular case I think it doesn't matter. Check:

Is it worth using Python's re.compile?

As you can see in the answer, and in the source code:

https://github.com/python/cpython/blob/master/Lib/re.py#L281

The implementation of the re module has a cache of the regular expression itself. So, the small speed up you see is probably because you avoid the lookup for the cache.

Now, as with the question, sometimes doing something like this is very relevant like, again, building a internal cache that remains namespaced to the function.

def heavy_processing(arg):
    return arg + 2

def myfunc(arg1):
    # Assign attribute to function if first call
    if not hasattr(myfunc, 'cache'):
        myfunc.cache = {}

    # Perform lookup in internal cache
    if arg1 in myfunc.cache:
        return myfunc.cache[arg1]

    # Very heavy and expensive processing with arg1
    result = heavy_processing(arg1)
    myfunc.cache[arg1] = result
    return result

And this is executed like this:

>>> myfunc.cache
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
AttributeError: 'function' object has no attribute 'cache'
>>> myfunc(10)
12
>>> myfunc.cache
{10: 12}

Answer 5

You can use a static function attribute to hold the compiled re. This example does something similar, keeping a translation table in one function attribute.

def static_var(varname, value):
    def decorate(func):
        setattr(func, varname, value)
        return func
    return decorate

@static_var("complements", str.maketrans('acgtACGT', 'tgcaTGCA'))
def rc(seq):
    return seq.translate(rc.complements)[::-1]