What is the best way to save and recall a sequence of functions and arguments

Question

In Python 3.8.8, I am looking for a way to save and recall a sequence of functions and associated arguments. I came across dataclasses which looks like a good way of doing this, but can't retrieve the function names or arguments. I'm not even sure if what I want to do is possible with dataclasses.

from dataclasses import dataclass
from dataclass_csv import DataclassWriter, DataclassReader

def foo(bar):
    print(bar)
    return True
   
@dataclass
class Step:
    name: str
    fn: any
    args: any = None

steps = [ Step(name ='step 1', fn=foo, args=('bar 1',)),
          Step(name ='step 2', fn=foo, args=('bar 2',)) ]

with open('steps.csv', "w") as f:
    w = DataclassWriter(f, steps, Step)
    w.write()
 
with open("steps.csv") as f:
    reader = DataclassReader(f, Step)
    rsteps  = [row for row in reader]
    print(rsteps)

steps.csv contains expected values.

name,fn,args
step 1,<function foo at 0x0000019E3A12EE50>,"('bar 1',)"
step 2,<function foo at 0x0000019E3A12EE50>,"('bar 2',)"

But when read back, rsteps ends up with booleans for fn and arg.

[Step(name='step 1', fn=True, args=True), Step(name='step 2', fn=True, args=True)]

I defined the fn and arg types as any because I couldn't find any type that would work there. I tried Callable[...,bool] (from typing import Callable) but got a TypeError: 'type' object is not subscriptable.

score 1 · Answer 1 · answered Jun 13 '21 at 04:50

I came up with this using lambda. It is reasonably simple and allows arbitrary arguments too.

from dataclasses import dataclass
from dataclass_csv import DataclassWriter, DataclassReader

def foo(bar,i):
    print(f'foo {bar} {i}')
    return True

@dataclass
class Step:
    name: str
    fn: str

steps = [ Step(name ='step 1', fn="foo('bar1', 1)"),
          Step(name ='step 2', fn="foo('bar2', 2)")]

with open('steps.csv', "w") as f:
    w = DataclassWriter(f, steps, Step)
    w.write()
    
reader = None 
with open("steps.csv") as f:
    reader = DataclassReader(f, Step)
    rsteps  = [row for row in reader]
    print(rsteps)

s=rsteps[0]
rfn = eval(f'lambda: {s.fn}')
rfn()

score 0 · Answer 2 · answered Jun 10 '21 at 13:26

any is a builtin funtion.
typing.Any is a type hint available in the typing library.
So you must not use any as a type hint, because it is a function, not a type.

Your foo function takes a string and returns a boolean, so its type is typing.Callable[[str], bool].

With these two modifications, we still write the same thing :

import typing
from dataclasses import dataclass
from dataclass_csv import DataclassWriter, DataclassReader


def foo(bar):
    print(bar)
    return True


@dataclass
class Step:
    name: str
    fn: typing.Callable[[str], bool]
    args: typing.Any = None


steps = [
    Step(name='step 1', fn=foo, args=('bar 1',)),
    Step(name='step 2', fn=foo, args=('bar 2',))
]

with open('steps.csv', "w") as f:
    w = DataclassWriter(f, steps, Step)
    w.write()

# with open("steps.csv") as f:
#     reader = DataclassReader(f, Step)
#     rsteps = [row for row in reader]
#     print(rsteps)

But the CSV file content cannot be read back, we get an error : TypeError: Type Callable cannot be instantiated; use a non-abstract subclass instead because it can't read the function back from what was written (<function foo at 0x000001EF2A232F28> for example).

Python functions can't be (de)serialized that way. Dataclasses are handy to store data, but they can't store functions without serious hacking.

You could marshall your function instead, take a look at this other question : Is there an easy way to pickle a python function (or otherwise serialize its code)?

What is the best way to save and recall a sequence of functions and arguments

2 Answers2