55

Is it possible to put a function in a data structure, without first giving it a name with def?

# This is the behaviour I want. Prints "hi".
def myprint(msg):
    print msg
f_list = [ myprint ]
f_list[0]('hi')
# The word "myprint" is never used again. Why litter the namespace with it?

The body of a lambda function is severely limited, so I can't use them.

Edit: For reference, this is more like the real-life code where I encountered the problem.

def handle_message( msg ):
    print msg
def handle_warning( msg ):
    global num_warnings, num_fatals
    num_warnings += 1
    if ( is_fatal( msg ) ):
        num_fatals += 1
handlers = (
    ( re.compile( '^<\w+> (.*)' ), handle_message ),
    ( re.compile( '^\*{3} (.*)' ), handle_warning ),
)
# There are really 10 or so handlers, of similar length.
# The regexps are uncomfortably separated from the handler bodies,
# and the code is unnecessarily long.

for line in open( "log" ):
    for ( regex, handler ) in handlers:
        m = regex.search( line )
        if ( m ): handler( m.group(1) )
David Moles
  • 48,006
  • 27
  • 136
  • 235
Tim
  • 13,904
  • 10
  • 69
  • 101
  • 1
    No it's not. `# The word "myprint" is never used again. Why litter the namespace with it?` Why do you invest so much time just to get rid of a single line that doesn't hurt you in any way? – phant0m Jul 08 '11 at 20:13
  • What is the purpose of doing this? – Udi Jul 08 '11 at 20:14
  • 2
    @phant0m, @Udi: I want my code to be pretty and easy to read. In real life, I've got a list of pairs of regexp-function pairs and run the function/handler on strings that match the regexp. The handlers are small enough to make definitions outside the list ugly and inappropriate. – Tim Jul 08 '11 at 20:18
  • I think it would help if you could provide your real-life example then. Maybe someone can provide you with a better solution if it's more clear what exactly you are doing. – phant0m Jul 08 '11 at 20:30
  • I think you will get nice answers if you post your actual problem. – Udi Jul 08 '11 at 20:44
  • 1
    I've added the real problem now. I usually dislike doing so, since it makes the question much more specific. I'll probably learn more from posting it, but not future visitors who've found the question by title. (ping @phant0m) – Tim Jul 08 '11 at 20:50
  • It's not too late to repost your addition as a new question (instead)! – Udi Jul 08 '11 at 20:52
  • 1
    Those function names are good documentation. If you were to make them anonymous the reader of your code would have to spend more brain cycles understanding what those functions do. – Steven Rumbalski Jul 08 '11 at 21:22
  • I added a solution to your real problem: http://stackoverflow.com/questions/6629876/how-to-make-an-anonymous-function-in-python-without-christening-it/6630703#6630703 – Udi Jul 08 '11 at 21:31
  • 1
    If you really want to these things, you maybe want to switch to perl. I know perl, but I use python for clarity's sake. You could model this Pattern properly, or hack it. In the latter case I dont think, tat the namespace pollution is your primary problem. – sleeplessnerd Jul 13 '11 at 15:36
  • 1
    @sleepless: I would prefer Perl, but it's not available at the target machine. – Tim Jul 13 '11 at 15:38

14 Answers14

39

This is based on Udi's nice answer.

I think that the difficulty of creating anonymous functions is a bit of a red herring. What you really want to do is to keep related code together, and make the code neat. So I think decorators may work for you.

import re

# List of pairs (regexp, handler)
handlers = []

def handler_for(regexp):
    """Declare a function as handler for a regular expression."""
    def gethandler(f):
        handlers.append((re.compile(regexp), f))
        return f
    return gethandler

@handler_for(r'^<\w+> (.*)')
def handle_message(msg):
    print msg

@handler_for(r'^\*{3} (.*)')
def handle_warning(msg):
    global num_warnings, num_fatals
    num_warnings += 1
    if is_fatal(msg):
        num_fatals += 1
Community
  • 1
  • 1
Gareth Rees
  • 64,967
  • 9
  • 133
  • 163
  • 2
    This is a pretty sweet "inside out" solution that does a good job of changing the way you think about the problem. – kindall Jul 08 '11 at 22:27
  • 1
    the saga continues: http://stackoverflow.com/questions/6629876/how-to-make-an-anonymous-function-in-python-without-christening-it/6635029#6635029 – Udi Jul 09 '11 at 14:11
16

Nicer DRY way to solve your actual problem:

def message(msg):
    print msg
message.re = '^<\w+> (.*)'

def warning(msg):
    global num_warnings, num_fatals
    num_warnings += 1
    if ( is_fatal( msg ) ):
        num_fatals += 1
warning.re = '^\*{3} (.*)'

handlers = [(re.compile(x.re), x) for x in [
        message,
        warning,
        foo,
        bar,
        baz,
    ]]
Udi
  • 29,222
  • 9
  • 96
  • 129
  • Much nicer than my attempt. I really should read about the available data structures before coming up with more ideas. Thanks! – Tim Jul 08 '11 at 21:33
15

Continuing Gareth's clean approach with a modular self contained solution:

import re

# in util.py
class GenericLogProcessor(object):

    def __init__(self):
      self.handlers = [] # List of pairs (regexp, handler)

    def register(self, regexp):
        """Declare a function as handler for a regular expression."""
        def gethandler(f):
            self.handlers.append((re.compile(regexp), f))
            return f
        return gethandler

    def process(self, file):
        """Process a file line by line and execute all handlers by registered regular expressions"""
        for line in file:
            for regex, handler in self.handlers:
                m = regex.search(line)
                if (m):
                  handler(m.group(1))      

# in log_processor.py
log_processor = GenericLogProcessor()

@log_processor.register(r'^<\w+> (.*)')
def handle_message(msg):
    print msg

@log_processor.register(r'^\*{3} (.*)')
def handle_warning(msg):
    global num_warnings, num_fatals
    num_warnings += 1
    if is_fatal(msg):
        num_fatals += 1

# in your code
with open("1.log") as f:
  log_processor.process(f)
Community
  • 1
  • 1
Udi
  • 29,222
  • 9
  • 96
  • 129
13

If you want to keep a clean namespace, use del:

def myprint(msg):
    print msg
f_list = [ myprint ]
del myprint
f_list[0]('hi')
inspectorG4dget
  • 110,290
  • 27
  • 149
  • 241
Udi
  • 29,222
  • 9
  • 96
  • 129
9

As you stated, this can't be done. But you can approximate it.

def create_printer():
  def myprint(x):
    print x
  return myprint

x = create_printer()

myprint is effectively anonymous here, since the variable scope in which it was created is no longer accessible to the caller. (See closures in Python.)

robert
  • 33,242
  • 8
  • 53
  • 74
  • I don't know how I feel about that last line. Isn't the entire point of closures in python that the scope persists _inside_ the closure? That being said, I like this solution a lot. – Wilduck Jul 08 '11 at 20:29
6

If you're concerned about polluting the namespace, create your functions inside of another function. Then you're only "polluting" the local namespace of the create_functions function and not the outer namespace.

def create_functions():
    def myprint(msg):
        print msg
    return [myprint]

f_list = create_functions()
f_list[0]('hi')
FogleBird
  • 74,300
  • 25
  • 125
  • 131
  • 2
    In addition to the namespace pollution, I'm also bothered by 1. using a temporary name and 2. having to define it in a different place when it's only going to be used once. Even though this solves the namespace problem, it only worsens the other problems. – Tim Jul 08 '11 at 20:54
5

You should not do it cause eval is evil, but you can compile function code on run time using FunctionType and compile:

>>> def f(msg): print msg
>>> type(f)
 <type 'function'>
>>> help(type(f))
...
class function(object)
 |  function(code, globals[, name[, argdefs[, closure]]])
 |
 |  Create a function object from a code object and a dictionary.
 |  The optional name string overrides the name from the code object.
 |  The optional argdefs tuple specifies the default argument values.
 |  The optional closure tuple supplies the bindings for free variables.    
...

>>> help(compile)
Help on built-in function compile in module __builtin__:

compile(...)
    compile(source, filename, mode[, flags[, dont_inherit]]) -> code object

    Compile the source string (a Python module, statement or expression)
    into a code object that can be executed by the exec statement or eval().
    The filename will be used for run-time error messages.
    The mode must be 'exec' to compile a module, 'single' to compile a
    single (interactive) statement, or 'eval' to compile an expression.
    The flags argument, if present, controls which future statements influence
    the compilation of the code.
    The dont_inherit argument, if non-zero, stops the compilation inheriting
    the effects of any future statements in effect in the code calling
    compile; if absent or zero these statements do influence the compilation,
    in addition to any features explicitly specified.
Udi
  • 29,222
  • 9
  • 96
  • 129
  • Good idea. Out of interest, what is the moral difference between `eval` and `compile`? – Tim Jul 08 '11 at 21:14
  • @Tim: http://stackoverflow.com/questions/2220699/whats-the-difference-between-eval-exec-and-compile-in-python – Mark Fowler May 17 '15 at 02:10
  • **Fantastic answer.** This novel approach is nothing less than a robust mechanism for dynamically defining Python callables (e.g., functions, methods) at runtime, complete with closure support! I've been grepping for something _exactly_ like this for longer than I care to admit. Thanks for the putting the puzzle pieces together, [Udi](https://stackoverflow.com/users/57952/udi). – Cecil Curry Jun 25 '16 at 07:08
3

The only way to make an anonymous function is with lambda, and as you know, they can only contain a single expression.

You can create a number of functions with the same name so at least you don't have to think of new names for each one.

It would be great to have truly anonymous functions, but Python's syntax can't support them easily.

Ned Batchelder
  • 364,293
  • 75
  • 561
  • 662
  • 1
    "but Python's syntax can't support them easily." can you elaborate on this? (or link to resources) Thanks. – phant0m Jul 08 '11 at 20:12
  • 3
    @phant0m: The model of indentation for block delimiting doesn't support starting/ending blocks inside expressions (e.g. inside parens), and it seems this can't be added easily (either very complex to implement or make indentation in *all* expression significant, which is breaks multi-line expressions). Therefore, you can't allow multiple statements in a lambda expression as you wouldn't know where is ended (no DEDENT token). I bet there's some more materal on the mailing list. –  Jul 08 '11 at 20:21
  • Ah yes, that makes perfect sense! I didn't really think as much to get to think about where and how they would actually be used and what it then would mean :) Thanks! – phant0m Jul 08 '11 at 20:27
  • 1
    I couldn't find a good description, but it has been rehashed many times, and @delnan is right: the indentation-based syntax can't accommodate statements in expressions. – Ned Batchelder Jul 08 '11 at 20:30
  • In contrast, the Scheme-derived R allows one to do: function(x,y,z) {w<- y+z ; x+w} where, for the uninitiated, "function" is the keyword that stands in for Python's "lambda". The main reason to use anonymous functions like this is to pass them as parameters to other functions. –  Jul 22 '15 at 22:29
3

As all said lambda is the only way, but you have to think not about lambda limitations but how to avoid them - for example you can use lists, dicts, comprehensions and so on in order to do what you want:

funcs = [lambda x,y: x+y, lambda x,y: x-y, lambda x,y: x*y, lambda x: x]
funcs[0](1,2)
>>> 3
funcs[1](funcs[0](1,2),funcs[0](2,2))
>>> -1
[func(x,y) for x,y in zip(xrange(10),xrange(10,20)) for func in funcs]

EDITED with print(try to look at the pprint module) and control-flow:

add = True
(funcs[0] if add else funcs[1])(1,2)
>>> 3

from pprint import pprint
printMsg = lambda isWarning, msg: pprint('WARNING: ' + msg) if isWarning else pprint('MSG:' + msg)
Artsiom Rudzenka
  • 27,895
  • 4
  • 34
  • 52
3

Python really, really doesn't want to do this. Not only does it not have any way to define a multi-line anonymous function, but function definitions don't return the function, so even if this were syntactically valid...

mylist.sort(key=def _(v):
                    try:
                        return -v
                    except:
                        return None)

... it still wouldn't work. (Although I guess if it were syntactically valid, they'd make function definitions return the function, so it would work.)

So you can write your own function to make a function from a string (using exec of course) and pass in a triply-quoted string. It's kinda ugly syntactically, but it works:

def function(text, cache={}):

    # strip everything before the first paren in case it's "def foo(...):"
    if not text.startswith("("):
        text = text[text.index("("):]

    # keep a cache so we don't recompile the same func twice
    if text in cache:
        return cache[text]

    exec "def func" + text
    func.__name__ = "<anonymous>"

    cache[text] = func
    return func

    # never executed; forces func to be local (a tiny bit more speed)
    func = None

Usage:

mylist.sort(key=function("""(v):
                                try:
                                    return -v
                                except:
                                    return None"""))
kindall
  • 178,883
  • 35
  • 278
  • 309
  • Except for syntax highlightning, I don't find the custom function with triple-quote ugly at all. I didn't know about the `func = None` trick -- where is this documented? – Tim Jul 08 '11 at 22:13
  • If you assign to a variable in a function, that variable is local. It's determined at compile time, not at execution, so it can be out of the actual path of execution. See: http://docs.python.org/reference/executionmodel.html, specifically "If a name is bound in a block, it is a local variable of that block." – kindall Jul 08 '11 at 22:18
  • Speed advantage of local variables: http://wiki.python.org/moin/PythonSpeed/PerformanceTips#Local_Variables I figured if I was gonna do something so hacky I'd at least make it as fast as possible. :-) – kindall Jul 08 '11 at 22:21
2

Personally I'd just name it something use it and not fret about it "hanging around". The only thing you'll gain by using suggestions such as as redefining it later or using del to drop the name out of the namespace is a potential for confusion or bugs if someone later comes along and moves some code around without groking what you're doing.

John Gaines Jr.
  • 11,174
  • 1
  • 25
  • 25
  • In addition to the namespace litter, I'm also bothered by 1. using a temporary name and 2. having to define it in a different place when it's only going to be used once. – Tim Jul 08 '11 at 20:53
2

You could use exec:

def define(arglist, body):
    g = {}
    exec("def anonfunc({0}):\n{1}".format(arglist,
                                     "\n".join("    {0}".format(line)
                                               for line in body.splitlines())), g)
    return g["anonfunc"]

f_list = [define("msg", "print(msg)")]
f_list[0]('hi')
codeape
  • 97,830
  • 24
  • 159
  • 188
1

The only option is to use a lambda expression, like you mention. Without that, it is not possible. That is the way python works.

thunderflower
  • 181
  • 1
  • 5
  • In this case, or cases where I'd need two statements, lambda functions wouldn't work, right? – Tim Jul 08 '11 at 20:10
  • Exactly. Since print isn't a statement it wont work. And you can only have one statement in a lambda. You can, however, make rather long statements with the and operator. But I guess that's not really what you're looking for. – thunderflower Jul 08 '11 at 20:12
  • 1
    @thunderflower: you can't have any statements in a lambda, only an expression. – Ned Batchelder Jul 08 '11 at 20:31
  • Yeah. Sorry about that. Expression is what I meant. – thunderflower Jul 08 '11 at 20:33
-1

If your function is complicated enough to not fit in a lambda function, then, for readability's sake, it would probably be best to define it in a normal block anyway.

TorelTwiddler
  • 5,996
  • 2
  • 32
  • 39
  • Would you consider "print x" complicated? – Tim Jul 08 '11 at 20:55
  • @Tim Nordenfur: He specifically said "The body of a lambda function is severely limited, so I can't use them." I assumed (perhaps incorrectly) that the print statement was an example. – TorelTwiddler Jul 08 '11 at 20:58
  • While there's surely a lot of too-long anonymous functions, I'm certain that there are some examples where a lambda-like syntax would be fine, even though it's too "complicated". `print x` is one counter-example, as is any statement that is not an expression. – Tim Jul 08 '11 at 21:03
  • `print` can be used as a function by using `from __future__ import print_statement`. `if` statements can be handled by `lambda`. If he's needs to use something like a `try...except`, then defining a block for it is a good idea. Most other statements are either built in to `lambda` or useless to lambda function. Perhaps `exec` would work as a counter-example? – TorelTwiddler Jul 08 '11 at 21:18