1

I'm writing a wrapper or pipeline to create a tfrecords dataset to which I would like to supply a function to apply to the dataset.

I would like to make it possible for the user to inject a function defined in another python file which is called in my script to transform the data.

Why? The only thing the user has to do is write the function which brings his data into the right format, then the existing code does the rest.

I'm aware of the fact that I could have the user write the function in the same file and call it, or to have an import statement etc.

So as a minimal example, I would like to have file y.py

def main(argv):
    # Parse args etc, let's assume it is there.

    dataset = tf.data.TFRecordDataset(args.filename)
    dataset = dataset.map(args.function)
    # Continue with doing stuff that is independent from actual content

So what I'd like to be able to do is something like this

    python y.py --func x.py my_func

And use the function defined in x.py my_func in dataset.map(...)

Is there a way to do this in python and if yes, which is the best way to do it?

hechth
  • 95
  • 11
  • So you have a function my_func defined in x.py? Why not `from x import my_func` and then call `my_func(...)` as you normally would with a locally defined function? Or `import x` then call `x.my_func(...)`? –  Oct 25 '19 at 12:15
  • @JustinEzequiel As mentioned in the question, I'm aware of this solution, but I don't want to use this, but to be able to pass the function to calling the script. – hechth Oct 25 '19 at 12:18
  • 1
    Then have you seen [importlib.import_module](https://docs.python.org/3/library/importlib.html#importlib.import_module)? It's your use-case that I don't get. –  Oct 25 '19 at 12:30
  • "I'm writing a wrapper or pipeline to create a tfrecords dataset to which I would like to supply a function to apply to the dataset." This is my use case. "Is there a way to do this in python and if yes, which is the best way to do it?" This is the question. So your answer is, there is no way to do it or rather there is no scenario in which this makes sense? – hechth Oct 25 '19 at 12:36

2 Answers2

1
  1. Pass the name of the file as an argument to your script (and function name)
  2. Read the file into a string, possibly extracting the given function
  3. use Python exec() to execute the code

An example:

file = "def fun(*args): \n  return args"
func = "fun(1,2,3)"

def execute(func, file):
    program = file + "\nresult = " + func
    local = {}
    exec(program, local)
    return local['result']

r = execute(func, file)
print(r)

Similar to here however we must use locals() as we are not calling exec in global scope.

Note: the use of exec is somewhat dangerous, you should be sure that the function is safe - if you are using it then its fine!

Hope this helps.

BenedictWilkins
  • 1,173
  • 8
  • 25
  • 1
    Since exec returns None, how can it serve as a function which is expected to return the transformed element? Or how can it return whatever that specified function returns? – hechth Oct 25 '19 at 12:16
  • 1
    it is discussed [here](https://stackoverflow.com/questions/23917776/how-do-i-get-the-return-value-when-using-python-exec-on-the-code-object-of-a-fun). There may be a better way than to use `exec` for sure! – BenedictWilkins Oct 25 '19 at 12:38
  • Okay so I can make it work by using another wrapper function, this would be a solution which exactly solves the problem specified in the question. So I'm waiting for more feedback but if nothing better comes up this can be an accepted answer if you edit it slightly to contain this info as well ;) – hechth Oct 25 '19 at 12:44
  • 1
    @hechth I have updated with an example. I hope this is what you were after. – BenedictWilkins Oct 25 '19 at 13:48
0

Ok so I have composed the answer myself now using the information from comments and this answer.

    import importlib, inspect, sys, os
    # path is given path to file, funcion_name is name of function and args are the function arguments

    # Create package and module name from path
    package = os.path.dirname(path).replace(os.path.sep,'.')
    module_name = os.path.basename(path).split('.')[0]

    # Import module and get members
    module = importlib.import_module(module_name, package)
    members = inspect.getmembers(module)

    # Find matching function
    function = [t[1] for t in members if t[0] == function_name][0]
    function(args)

This exactly solves the question, since I get a callable function object which I can call, pass around, use it as a normal function.

hechth
  • 95
  • 11