1

How to use the data set without my_compute_function. From file1 in repository, I want to call a function which is defined in another file. In the second file, I want to make use of the data set, my_input_integration, may be without my_compute_function. How to combine the datasets from two different repository files. I do not want to combine in one file because I want to use 2nd file as utility file. It would be great if anyone can answer this.

Repository File 1

from transforms.api import transform, Input, Output


@transform(
    my_output=Output("/my/output"),
    my_input=Input("/my/input"),
)
def my_compute_function(my_input, my_output):
    return calling_function(my_input, my_output)

Repository File 2

from transforms.api import transform, Input, Output


@transform(
    my_input_integration =Input("/my/input"),
)
def calling_function(my_input, my_output, my_input_integration??)
   
    return my_output.write_dataframe(
        my_input.dataframe(),
        column_descriptions=my_dictionary
    )
mpSchrader
  • 902
  • 3
  • 20
Ram D
  • 167
  • 1
  • 8

1 Answers1

4

If I understand correctly what you're trying to achieve, you can't directly do that -- any input to a transform has to be defined in that transform, and then passed into the utility function, you can't "inject" inputs.

So the most straightforward way to achieve what you want would be to do something like this:

file 1:

@transform(
    my_output=Output("/my/output"),
    my_input=Input("/my/input"),
    my_input_integration=Input("/my/input_integration"),
)
def my_compute_function(my_input, my_output, my_input_integration):
    return calling_function(my_input, my_output, my_input_integration)

file 2:

def calling_function(my_input, my_output, my_input_integration)
    return my_output.write_dataframe(
        my_input.dataframe(),
        column_descriptions=my_dictionary
    )

If you really think you need the ability to automatically "inject" datasets, and adding them as parameters to your transform would be too cumbersome, it would be possible to take a more sophisticated approach where you define a custom wrapper that you apply to the transform function that makes the input datasets automatically available. I would really avoid this though, as it adds a lot of complexity and "magic" to the code that will be hard to understand for a newcomer and casual reader.

mpSchrader
  • 902
  • 3
  • 20
Jonathan Ringstad
  • 967
  • 5
  • 9
  • yes, you understood correctly. Appreciate you for putting in your thoughts. It helps. however, I would really like to see and learn how that wrapper looks like. – Ram D Oct 13 '20 at 13:02
  • @RamD I don't know from the top of my head what it would look like, but check out https://realpython.com/primer-on-python-decorators/ for an introduction on how python decorators work. From there, what you'd probably do is to either replace the `@transform` decorator with your own `@transform_with_integration` decorator, or to stack them, so that you put your custom `@with_integration` decorator after the `@transform` decorator. From within this decorator, you then somehow make the newly added input available (e.g. as parameter or otherwise.) Though, I'd not recommend going down this route. – Jonathan Ringstad Oct 13 '20 at 13:28