12

Context

Suppose I have the following Python code:

def example_function(numbers, n_iters):
    sum_all = 0
    for number in numbers:
        for _ in range(n_iters):
            number = halve(number)
        sum_all += number
    return sum_all


ns = [1, 3, 12]
print(example_function(ns, 3))

example_function here is simply going through each of the elements in the ns list and halving them 3 times, while accumulating the results. The output of running this script is simply:

2.0

Since 1/(2^3)*(1+3+12) = 2.

Now, let's say that (for any reason, perhaps debugging, or logging), I would like to display some type of information about the intermediate steps that the example_function is taking. Maybe I would then rewrite this function into something like this:

def example_function(numbers, n_iters):
    sum_all = 0
    for number in numbers:
        print('Processing number', number)
        for i_iter in range(n_iters):
            number = number/2
            print(number)
        sum_all += number
        print('sum_all:', sum_all)
    return sum_all

which now, when called with the same arguments as before, outputs the following:

Processing number 1
0.5
0.25
0.125
sum_all: 0.125
Processing number 3
1.5
0.75
0.375
sum_all: 0.5
Processing number 12
6.0
3.0
1.5
sum_all: 2.0

This achieves exactly what I intended. However, this goes a bit against the principle that a function should only do one thing, and now the code for example_function is sligthly longer and more complex. For such a simple function this is not a problem, but in my context I have quite complicated functions calling each other, and the printing statements often involve more complicated steps than shown here, resulting in a substantial increase in complexity of my code (for one of my functions there were more lines of code related to logging than there were lines related to its actual purpose!).

Furthermore, if I later decide that I don't want any printing statements in my function anymore, I would have to go through example_function and delete all of the print statements manually, along with any variables related this functionality, a process which is both tedious and error-prone.

The situation gets even worse if I would like to always have the possibility of printing or not printing during the function execution, leading me to either declaring two extremely similar functions (one with the print statements, one without), which is terrible for maintaining, or to define something like:

def example_function(numbers, n_iters, debug_mode=False):
    sum_all = 0
    for number in numbers:
        if debug_mode:
            print('Processing number', number)
        for i_iter in range(n_iters):
            number = number/2
            if debug_mode:
                print(number)
        sum_all += number
        if debug_mode:
            print('sum_all:', sum_all)
    return sum_all

which results in a bloated and (hopefully) unnecessarily complicated function, even in the simple case of our example_function.

Question

Is there a pythonic way to "decouple" the printing functionality from the original functionality of the example_function?

More generally, is there a pythonic way to decouple optional functionality from a function's main purpose?

What I have tried so far:

The solution I have found at the moment is using callbacks for the decoupling. For instance, one can rewrite the example_function like this:

def example_function(numbers, n_iters, callback=None):
    sum_all = 0
    for number in numbers:
        for i_iter in range(n_iters):
            number = number/2

            if callback is not None:
                callback(locals())
        sum_all += number
    return sum_all

and then defining a callback function that performs whichever printing functionality I want:

def print_callback(locals):
    print(locals['number'])

and calling example_function like this:

ns = [1, 3, 12]
example_function(ns, 3, callback=print_callback)

which then outputs:

0.5
0.25
0.125
1.5
0.75
0.375
6.0
3.0
1.5
2.0

This successfully decouples the printing functionality from the base functionality of example_function. However, the main problem with this approach is that the callback function can only be run at a specific part of the example_function (in this case right after halving the current number), and all of the printing has to happen exactly there. This sometimes forces the design of the callback function to be quite complicated (and makes some behaviors impossible to achieve).

For instance, if one would like to achieve exactly the same type of printing as I did in a previous part of the question (showing which number is being processed, along with its corresponding halvings) the resulting callback would be:

def complicated_callback(locals):
    i_iter = locals['i_iter']
    number = locals['number']
    if i_iter == 0:
        print('Processing number', number*2)
    print(number)
    if i_iter == locals['n_iters']-1:
        print('sum_all:', locals['sum_all']+number)

which results in exactly the same output as before:

Processing number 1.0
0.5
0.25
0.125
sum_all: 0.125
Processing number 3.0
1.5
0.75
0.375
sum_all: 0.5
Processing number 12.0
6.0
3.0
1.5
sum_all: 2.0

but is a pain to write, read, and debug.

James
  • 32,991
  • 4
  • 47
  • 70
JLagana
  • 1,224
  • 4
  • 14
  • 33
  • 6
    check out the python `logging` module – Chris_Rands Oct 24 '19 at 15:05
  • @Chris_Rands is right .. use the logging module.. that way you can turn logging on and off .. use the following link . https://stackoverflow.com/questions/2266646/how-to-i-disable-and-re-enable-console-logging-in-python – Yatish Kadam Oct 24 '19 at 15:08
  • 2
    I don't see how the `logging` module would help here. Although my question uses `print` statements when setting up the context, I'm actually looking for a solution of how to decouple any type of optional functionality from a function's main purpose. For instance, maybe I want a function to plot things as it runs. In that case I believe that the `logging` module wouldn't even be applicable. – JLagana Oct 24 '19 at 15:12
  • Also, even if we're just talking about printing functionality, wouldn't you need to add lines like `logging.info(number)` to the `example_function`? Although this would probably help if one wants optional print statements, it wouldn't solve the main issue, which is that the code for `example_function` would have to perform both its main functionality *and* printing, resulting in a longer and more complex function. – JLagana Oct 24 '19 at 15:21
  • Maybe you should do this: store a value in a config file, and inside your function write like this: `if CONFIG.DEBUG_ENABLED: print(number)`. Alternatively you can use command-line arguments instead of config file. Can't imagine other options to do something flexible, not just printing/logging. – sanyassh Oct 24 '19 at 15:36
  • 3
    @Pythonic is an adjective that describes python syntax/style/structure/usage to upheld the philosophy of Python. This is not a syntactic or design rule, rather an approach that needs to be uphold responsibly to produce a clean and maintainable python codebase. In your case, having few lines of trace or print statements adds values to maintainability then have it; don’t be hard on your self. Consider any of aforementioned approach which you think is ideal. – S.N Oct 24 '19 at 15:39
  • @Nair None of the approaches I mentioned have good maintainability in my opinion. This is why I asked the question. – JLagana Oct 24 '19 at 15:46
  • @JLagana, Assuming the trace steps here are helping hands for you.With this in mind, complicated_callback(locals) is a good option. I know you mentioned about pain to write. But remember how often you change this. One other solution that I think worth looking is the 'decorator pattern' style of logging which extends from the complicated_callback(locals) approach. – S.N Oct 24 '19 at 15:54
  • 1
    This question is too broad. We might be able to address specific questions (as the suggestions to use `logging` demonstrate), but not how to separate arbitrary code. – chepner Oct 31 '19 at 13:00
  • Are the incremental outputs intended to provide info for the users of this function or for you who is defining it? If they are for the users, then write conditional print statements. If not, then use a debugger (e.g. [ipdb](https://pypi.org/project/ipdb/)) – j-i-l Nov 01 '19 at 12:15

4 Answers4

4

If you need functionality outside the function to use data from inside the function, then there needs to be some messaging system inside the function to support this. There is no way around this. Local variables in functions are totally isolated from the outside.

The logging module is quite good at setting up a message system. It is not only restricted to printing out the log messages - using custom handlers, you can do anything.

Adding a message system is similar to your callback example, except that the places where the 'callbacks' (logging handlers) are handled can be specified anywhere inside the example_function (by sending the messages to the logger). Any variables that are needed by the logging handlers can be specified when you send the message (you can still use locals(), but it is best to explicitly declare the variables you need).

A new example_function might look like:

import logging

# Helper function
def send_message(logger, level=logging.DEBUG, **kwargs):
  logger.log(level, "", extra=kwargs)

# Your example function with logging information
def example_function(numbers, n_iters):
    logger = logging.getLogger("example_function")
    # If you have a logging system set up, then we don't want the messages sent here to propagate to the root logger
    logger.propagate = False
    sum_all = 0
    for number in numbers:
        send_message(logger, action="processing", number=number)
        for i_iter in range(n_iters):
            number = number/2
            send_message(logger, action="division", i_iter=i_iter, number=number)
        sum_all += number
        send_message(logger, action="sum", sum=sum_all)
    return sum_all

This specifies three locations where the messages could be handled. On its own, this example_function will not do anything other than the functionality of the example_function itself. It will not print out anything, or do any other functionality.

To add extra functionality to the example_function, then you will need to add handlers to the logger.

For example, if you want to do some printing out of the sent variables (similar to your debugging example), then you define the custom handler, and add it to the example_function logger:

class ExampleFunctionPrinter(logging.Handler):
    def emit(self, record):
        if record.action == "processing":
          print("Processing number {}".format(record.number))
        elif record.action == "division":
          print(record.number)
        elif record.action == "sum":
          print("sum_all: {}".format(record.sum))

example_function_logger = logging.getLogger("example_function")
example_function_logger.setLevel(logging.DEBUG)
example_function_logger.addHandler(ExampleFunctionPrinter())

If you want to plot the results on a graph, then just define another handler:

class ExampleFunctionDivisionGrapher(logging.Handler):
    def __init__(self, grapher):
      self.grapher = grapher

    def emit(self, record):
      if record.action == "division":
        self.grapher.plot_point(x=record.i_iter, y=record.number)

example_function_logger = logging.getLogger("example_function")
example_function_logger.setLevel(logging.DEBUG)
example_function_logger.addHandler(
    ExampleFunctionDivisionGrapher(MyFancyGrapherClass())
)

You can define and add in whatever handlers you want. They will be totally separate from the functionality of the example_function, and can only use the variables that the example_function gives them.

Although logging can be used as a messaging system, it might be better to move to a fully fledged messaging system, such as PyPubSub, so that it doesn't interfere with any actual logging that you might be doing:

from pubsub import pub

# Your example function
def example_function(numbers, n_iters):
    sum_all = 0
    for number in numbers:
        pub.sendMessage("example_function.processing", number=number)
        for i_iter in range(n_iters):
            number = number/2
            pub.sendMessage("example_function.division", i_iter=i_iter, number=number)
        sum_all += number
        pub.sendMessage("example_function.sum", sum=sum_all)
    return sum_all

# If you need extra functionality added in, then subscribe to the messages.
# Otherwise nothing will happen, other than the normal example_function functionality.
def handle_example_function_processing(number):
    print("Processing number {}".format(number))

def handle_example_function_division(i_iter, number):
    print(number)

def handle_example_function_sum(sum):
    print("sum_all: {}".format(sum))

pub.subscribe(
    "example_function.processing",
    handle_example_function_processing
)
pub.subscribe(
    "example_function.division",
    handle_example_function_division
)
pub.subscribe(
    "example_function.sum",
    handle_example_function_sum
)
RPalmer
  • 556
  • 3
  • 4
  • Thanks for the answer, RPalmer. The code you provided using the`logging` module is indeed more organized and maintainable than what I proposed using `print` and `if` statements. However, it does not decouple the printing functionality from the main functionality of the `example_function` function. That is, the main problem of having `example_function` do two things at once still remains, making its code more complicated than what I would like it to be. – JLagana Nov 05 '19 at 09:17
  • Compare this with for example my callback suggestion. Using callbacks, `example_function` now only has one functionality, and the printing stuff (or whatever other functionality we would like to have) happens outside of it. – JLagana Nov 05 '19 at 09:18
  • Hi @JLagana. My `example_function` is decoupled from the printing functionality - the only added functionality to the function is sending the messages. It is similar to your callback example, except that it only sends specific variables that you want, rather than all of the `locals()`. It is up to the log handlers (which you attach to the logger somewhere else) to do the extra functionality (printing, graphing, etc). You don't need to attach any handlers at all, in which case nothing will happen when the messages are sent. I have updated my post to make this more clear. – RPalmer Nov 05 '19 at 11:59
  • I stand corrected, your example did decouple the printing functionality from the main functionality of `example_function`. Thanks for making it extra clear now! I really like this answer, the only price being paid is the added complexity of passing messages, which, like you mentioned, seems to be unavoidable. Thanks also for the reference to PyPubSub, which led me to read up on the the [observer pattern](https://en.wikipedia.org/wiki/Observer_pattern). – JLagana Nov 05 '19 at 12:31
1

I have updated my answer with a simplification: function example_function is passed a single callback or hook with a default value such that example_function no longer needs to test to see whether it was passed or not:

hook=lambda *args, **kwargs: None

The above is a lambda expression that returns None and example_function can call this default value for hook with any combination of positional and keyword parameters at various places within the function.

In the example below, I am only interested in the "end_iteration" and "result" events.

def example_function(numbers, n_iters, hook=lambda *args, **kwargs: None):
    hook("init")
    sum_all = 0
    for number in numbers:
        for i_iter in range(n_iters):
            hook("start_iteration", number)
            number = number/2
            hook("end_iteration", number)
        sum_all += number
    hook("result", sum_all)
    return sum_all

if __name__ == '__main__':
    def my_hook(event_type, *args):
        if event_type in ["end_iteration", "result"]:
            print(args[0])

    print('sum = ', example_function([1, 3, 12], 3))
    print('sum = ', example_function([1, 3, 12], 3, my_hook))

Prints:

sum =  2.0
0.5
0.25
0.125
1.5
0.75
0.375
6.0
3.0
1.5
2.0
sum =  2.0

The hook function can be as simple or as elaborate as you want. Here it is doing a check of the event type and doing a simple print. But it could obtain a logger instance and log the message. You can have all the richness of logging if you need it but simplicity if you don't.

Booboo
  • 38,656
  • 3
  • 37
  • 60
  • Thanks for the answer, Ronald. The idea of extending the callback idea to execute callbacks in different parts of the function (and passing a context variable to them) seems like the best way to go, indeed. It makes it much easier to write callbacks and at a reasonable price in added complexity to `example_function`. – JLagana Nov 05 '19 at 14:29
  • Nice touch with the default value; it's a simple way to remove a lot of `if` statements :) – JLagana Nov 05 '19 at 14:31
1

You can define a function encapsulating the debug_mode condition and pass the desired optional function and its arguments to that function (as suggested here): 

def DEBUG(function, *args):
    if debug_mode:
        function(*args)

def example_function(numbers, n_iters):
    sum_all = 0
    for number in numbers:
        DEBUG(print, 'Processing number', number)
        for i_iter in range(n_iters):
            number = number/2
            DEBUG(print, number)
        sum_all += number
        DEBUG(print, 'sum_all:', sum_all)
    return sum_all

ns = [1, 3, 12]
debug_mode = True
print(example_function(ns, 3))

Note that debug_mode must obviously have been assigned a value before calling DEBUG.

It is of course possible to invoke functions other than print.

You could also extend this concept to several debug levels by using a numeric value for debug_mode.

Gerd
  • 2,568
  • 1
  • 7
  • 20
  • Thanks for the answer, Gerd. Indeed your solution gets rid of the need for `if` statements all over the place, and also makes it easier to turn printing on and off. However, it does not decouple the printing functionality from the main functionality of `example_function`. Compare this with for example my callback suggestion. Using callbacks, example_function now only has one functionality, and the printing stuff (or whatever other functionality we would like to have) happens outside of it. – JLagana Nov 05 '19 at 14:19
1

If you want to stick with just print statements, you can use a decorator that adds an argument which turns on/off the printing to console.

Here is a decorator that add the keyword-only argument and default value of verbose=False to any function, updates the docstring and signature. Calling the function as-is returns the expected output. Calling the function with verbose=True will turn on the print statements and return the expected output. This has the added benefit of not having to preface every print with a if debug: block.

from functools import wraps
from inspect import cleandoc, signature, Parameter
import sys
import os

def verbosify(func):
    @wraps(func)
    def wrapper(*args, **kwargs):
        def toggle(*args, verbose=False, **kwargs):
            if verbose:
                _stdout = sys.stdout
            else:
                _stdout = open(os.devnull, 'w')
            with redirect_stdout(_stdout):
                return func(*args, **kwargs)
        return toggle(*args, **kwargs)
    # update the docstring
    doc = '\n\nOption:\n-------\nverbose : bool\n    '
    doc += 'Turns on/off print lines in the function.\n '
    wrapper.__doc__ = cleandoc(wrapper.__doc__ or '\n') + doc
    # update the function signature to include the verbose keyword
    sig = signature(func)
    param_verbose = Parameter('verbose', Parameter.KEYWORD_ONLY, default=False)
    sig_params = tuple(sig.parameters.values()) + (param_verbose,)
    sig = sig.replace(parameters=sig_params)
    wrapper.__signature__ = sig
    return wrapper

Wrapping your function now allows you to turn on/off print functions using verbose.

@verbosify
def example_function(numbers, n_iters):
    sum_all = 0
    for number in numbers:
        print('Processing number', number)
        for i_iter in range(n_iters):
            number = number/2
            print(number)
        sum_all += number
        print('sum_all:', sum_all)
    return sum_all

Examples:

example_function([1,3,12], 3)
# returns:
2.0

example_function([1,3,12], 3, verbose=True)
# returns/prints:
Processing number 1
0.5
0.25
0.125
sum_all: 0.125
Processing number 3
1.5
0.75
0.375
sum_all: 0.5
Processing number 12
6.0
3.0
1.5
sum_all: 2.0
2.0

When you inspect example_function, you will see the updated documentation as well. Since your function doesn't have a docstring, it is just what is in the decorator.

help(example_function)
# prints:
Help on function example_function in module __main__:

example_function(numbers, n_iters, *, verbose=False)
    Option:
    -------
    verbose : bool
        Turns on/off print lines in the function.

In terms of coding philosophy. Having function that incur not side-effects is a functional programming paradigm. Python can be a functional language, but it is not designed to be exclusively that way. I always design my code with the user in mind.

If adding the option to print the calculation steps is a benefit to the user, then there is NOTHING wrong with do that. From a design standpoint, you are going to be stuck with adding in the print/logging commands somewhere.

James
  • 32,991
  • 4
  • 47
  • 70
  • Thanks for the answer, James. The provided code is indeed more organized and maintainable than the one I proposed, that uses `print` and `if` statements. Furthermore, it manages to actually decouple part of the functionality of printing from `example_function`'s main functionality, which was very nice (I also liked that the decorator automatically appends to the docstring, nice touch). However, it does not fully decouple the printing functionality from the main functionality of `example_function`: you still have to add the `print` statements and any accompanying logic to the function`s body. – JLagana Nov 05 '19 at 09:28
  • Compare this with for example my callback suggestion. Using callbacks, example_function now only has one functionality, and the printing stuff (or whatever other functionality we would like to have) happens outside of it. – JLagana Nov 05 '19 at 09:28
  • Lastly, we agree that if printing the calculation steps is a benefit to the user, then I'll be stuck with adding the printing commands somewhere. I however want them to be outside of the `example_function`'s body, so that it's complexity remains only associated to the complexity of its main functionality. In my real-life application of all of this, I have a main function which is already significantly complex. Adding printing/plotting/logging statements to its body makes it become a beast that has been quite challenging to maintain and debug. – JLagana Nov 05 '19 at 09:32