How to avoid repeating kwargs in Python?

Question

Suppose there is a method:

def train_model(self, out_dir='./out/',
                test_size=0.2, train_size=None,
                random_state=None, shuffle=True, stratify=None,
                epochs=DEFAULT_EPOCHS, batch_size=DEFAULT_BATCH_SIZE):
    ...
    self.model.train(test_size=test_size, train_size=train_size, random_state=random_state, shuffle=shuffle, stratify=stratify, epochs=epochs, batch_size=batch_size)

And inside this function another method with the same signature will be called, then I have to pass all the params manually. I don't want to use kwargs in train_model as it's a public method that may used by others, so I hope to keep the typing information. I don't know if there are methods to allow me to keep the typing information in kwargs of outer function.

The same functionality in TypeScript can be achieved using the Parameters utility types. For example,

function sum(a: int, b: int) {
    return a + b;
}

type SumParamsType = Paramters<typeof sum>

// Then you can use the SumPramsType in other places.

A failed example of Python:

from typing import TypeVar
T = TypeVar('T')

def f1(a=1, b=2, c=3):
    return a+b+c

# Is there anything like T=Parameters(type(f1)) in Python?

def f2(z=0, **kwargs: T):
    return z+f1(**kwargs)

# T cannot capture the kwargs of f1 (of course it won't)

And this doesn't works either:

def f1(a=1, b=2, c=3):
    return a+b+c

def f2(z=0, **kwargs: f1.__annotations__['kwargs']):
    return z + f1(**kwargs)

# kwargs has the type Any

Does this answer your question? [Type annotations for \*args and \*\*kwargs](https://stackoverflow.com/questions/37031928/type-annotations-for-args-and-kwargs) — MEE, Feb 19 '23 at 23:30
I am not sure. The solution it provides require Python 3.11. Besides what I want to know is not about how to annotate `**kwargs` but to reuse the type information of `self.model.train` in the `train_model`. I think that's different. — link89, Feb 20 '23 at 02:38
Why the example you provided is failed? The `f2` function does not return anything because `return` statement is missing. — Jurakin, Feb 20 '23 at 07:07
It's runnable, what I mean is T fail to capture the argument typing info of `f1`. — link89, Feb 20 '23 at 07:11
Maybe `f2.__annotations__` that returns `{'kwargs': ~T}` can be answer to one of your questions. [See docs](https://docs.python.org/3/howto/annotations.html). I don't get what do you mean by `T cannot capture kwargs of f1`. Can it capture args and kwargs not? — Jurakin, Feb 20 '23 at 07:31
Now I know what you mean. I don't see any other way to do it, except copying the function (won't work if you need to perform some operations). `self.train_model = self.model.train` — Jurakin, Feb 20 '23 at 08:44

score 1 · Answer 1 · answered Feb 20 '23 at 15:21

The closest you can get is to use a TypedDict with Unpack (available on Python < 3.11 via typing_extensions):

from typing_extensions import Unpack, TypedDict, NotRequired


class Params(TypedDict):
    a: NotRequired[int]
    b: NotRequired[int]
    c: NotRequired[int]


def f1(**kwargs: Unpack[Params]):
    a = kwargs.pop('a', 1)
    b = kwargs.pop('b', 1)
    c = kwargs.pop('c', 1)
    return a + b + c


def f2(z=0, **kwargs: Unpack[Params]):
    return z + f1(**kwargs)

Note that your IDE may not support Unpack if it does not use mypy --enable-incomplete-feature=Unpack. VSCode supports it out of the box. PyCharm, probably not.

If you control both function definitions, you may find it easier to change your methods to accept a dataclass encapsulating all the parameters and their defaults instead of taking each parameter individually.

I guess that's the best I can get for now. I am just wondering is there any PEP suggest some utility types like what typescript does: https://www.typescriptlang.org/docs/handbook/utility-types.html — link89, Feb 20 '23 at 15:30
There is a discussion about adding both a default catch-all type to `TypedDict` and also specifying defaults of each argument [here](https://github.com/python/mypy/issues/6131) but it doesn't seem that it has gotten anywhere yet. — MEE, Feb 20 '23 at 15:39

score 1 · Answer 2 · answered Feb 20 '23 at 15:30

You could create a class containing the training arguments and pass it to the train method, as is done in the HuggingFace Transformers library

Here is the code from their GitHub:

from dataclasses import asdict, dataclass, field, fields

#...

@dataclass
class TrainingArguments:
    framework = "pt"
    output_dir: str = field(
        metadata={"help": "The output directory where the model predictions and checkpoints will be written."},
    )
    overwrite_output_dir: bool = field(
        default=False,
        metadata={
            "help": (
                "Overwrite the content of the output directory. "
                "Use this to continue training if output_dir points to a checkpoint directory."
            )
        },
    )

    do_train: bool = field(default=False, metadata={"help": "Whether to run training."})
    do_eval: bool = field(default=False, metadata={"help": "Whether to run eval on the dev set."})
    do_predict: bool = field(default=False, metadata={"help": "Whether to run predictions on the test set."})
    evaluation_strategy: Union[IntervalStrategy, str] = field(
        default="no",
        metadata={"help": "The evaluation strategy to use."},
    )
    prediction_loss_only: bool = field(
        default=False,
        metadata={"help": "When performing evaluation and predictions, only returns the loss."},
    )

    per_device_train_batch_size: int = field(
        default=8, metadata={"help": "Batch size per GPU/TPU core/CPU for training."}
    )
    per_device_eval_batch_size: int = field(
        default=8, metadata={"help": "Batch size per GPU/TPU core/CPU for evaluation."}
    )
    # ...

It is a bit verbose but very clear and will work with your IDE type hinting.

Minh-Long Luu · Answer 3 · 2023-02-20T11:46:33.697

0

You will need a combination of locals() and .__code__.co_varnames:

def f2(b=5, c=7):
    return b*c

def f1(a=1, b=2, c=3):
    sss = locals().copy()
    f2_params = f2.__code__.co_varnames
    return f2(**{x:y for x, y in sss.items() if x in f2_params})

print(f1())
>>> 6

Edit

If you want to use **kwargs, try this:

def f2(b=5, c=7):
    return b*c

def f1(a=1, **kwargs):
    sss = locals()['kwargs'].copy()
    f2_params = f2.__code__.co_varnames
    return f2(**{x:y for x, y in sss.items() if x in f2_params})

print(f1(b=10, c=3))

edited Feb 20 '23 at 11:46

answered Feb 20 '23 at 09:07

Minh-Long Luu

2,393
1
17
39

It's about type annotation and DRY priciple. Instead of `def f1(a=1, b=2, c=3)`, I hope I could write `def f1(a=1, **kwargs)`, and kwargs should keep the type infomation of f2's input arguments. – link89 Feb 20 '23 at 09:10
It's the same as my second failed example, the type of kwargs is Any. The key point is to have kwargs carry the type information of f2. – link89 Feb 20 '23 at 14:43

How to avoid repeating kwargs in Python?

3 Answers3

Edit