0

I have a piece of code that generates a function from many smaller functions while making the outermost one accept an argument x.

In other words, I have an input x and I need to do various transformations to x that are decided at runtime.

This is done by iteratively calling this function (it essentially wraps a function in another function).

Here is the function:

def build_layer(curr_layer: typing.Callable, prev_layer: Union[typing.Callable, int]) -> typing.Callable:

    def _function(x):
        return curr_layer(prev_layer(x) if callable(prev_layer) else x)

    return _function

Sidenote: as you can see if prev_layer is not callable it gets substituted for input x so I am using dummy integers to indicate where input goes.

The problem: this code cannot be pickled. I do not seem to be able to figure out a way to rewrite this code in such a way to be pickleable.

Note: I need this object to be persisted on disk, but also its used in multiprocessing where its pickled for IPC (these functions are not used there, so technically they could be moved)

I have also a more complex version of this function that handles multiple inputs (using fixed aggregation function, in this case torch.cat) I know these two can be merged into one generic function and I will do that once I get it to work.

Here is the code for the second function:

def build_layer_multi_input(curr_layer: typing.Callable, prev_layers: list) -> typing.Callable:

    def _function(x):
        return curr_layer(torch.cat([layer(x) if callable(layer) else x for layer in prev_layers]))

    return _function
Igor Vereš
  • 33
  • 5
  • how are the functions that are passing as layers defined? Like I could imagine a class that just has a list of functions to implement the same thing but the callable things would still need to be pickled – Tadhg McDonald-Jensen Sep 02 '21 at 20:08
  • The layers are callable classes themselves that do the transformation. – Igor Vereš Sep 02 '21 at 20:14
  • can you confirm they are pickleable? If they are then I can offer one solution but if not I'd need to suggest `marshal`. – Tadhg McDonald-Jensen Sep 02 '21 at 20:27
  • Do you mean the result of *calling* `build_layer` is not pickleable? The function itself is. Unrelated, but I would not let the second argument be a non-callable. Let the caller provide a constant function instead. – chepner Sep 02 '21 at 20:27
  • 1
    The `pickle` module doesn't pickle code, so there's a basic misunderstanding going on here. – martineau Sep 02 '21 at 20:27
  • (That is, and ignoring the fact that anonymous functions aren't pickleable, `build_layer(f, lambda: 3)` instead of `build_layer(f, 3)`.) – chepner Sep 02 '21 at 20:34
  • (Actually, you ignore `prev_layer` if it isn't callable, rather than using it as an initial value. You could just return `curr_layer` when `prev_layer` isn't callable, rather than defining a new function that just passes its argument on to `curr_layer`.) – chepner Sep 02 '21 at 20:37
  • You are in fact correct I misunderstood how pickle works, the *result* of these functions is not pickleable. But the problem still stands. `.build_layer_multi_input.._function at 0x0000028E4BA35820>` – Igor Vereš Sep 02 '21 at 20:42
  • @chepner this gets more complicated in case are multiple inputs (second function). I don't think I could identity function, due to this being tensors pytorch - there is backend machinery that tracks what happens to tensors for calculating gradient later. I am not sure though I would have to try. – Igor Vereš Sep 02 '21 at 20:47
  • The problem here isn't nested functions, but `TorchPhenotype`. – chepner Sep 02 '21 at 20:49
  • Functions are pickleable, but that doesn't mean everything that's callable is. – chepner Sep 02 '21 at 20:50

1 Answers1

0

I resolved this by attaching the return value of these functions to a class instance as described in this thread.

Igor Vereš
  • 33
  • 5