python 3 typing varadic "apply" style definitions

Question

I've been struggling to write "varadic" argument lists type definitions.

for example, giving types to:

def foo(fn, *args):
    return fn(*args)

the best I've been able to do is using a suggestion from here:

from typing import overload, Callable, TypeVar

A = TypeVar('A')
B = TypeVar('B')
C = TypeVar('C')
R = TypeVar('R')

@overload
def foo(fn: Callable[[A], R], a: A) -> R: ...
@overload
def foo(fn: Callable[[A, B], R], a: A, b: B) -> R: ...
@overload
def foo(fn: Callable[[A, B, C], R], a: A, b: B, c: C) -> R: ...

def foo(fn, *args):
    return fn(*args)

which mostly does the right thing… for example, given:

def bar(i: int, j: int) -> None:
    print(i)

the following succeeds:

foo(bar, 10, 12)

while these fail:

foo(bar, 10)
foo(bar, 10, 'a')
foo(bar, 10, 12) + 1

but if I check with mypy --strict I get:

test.py:15: error: Function is missing a type annotation

(which is saying that the final foo definition doesn't have any types itself)

I can redefine foo to be:

def foo(fn: Callable[..., R], *args: Any) -> R:
    return fn(*args)

but then when I run mypy --strict I get:

test.py:15: error: Overloaded function implementation does not accept all possible arguments of signature 1
test.py:15: error: Overloaded function implementation does not accept all possible arguments of signature 2
test.py:15: error: Overloaded function implementation does not accept all possible arguments of signature 3

which I don't really understand.

if anyone can suggest a better way of giving types to this sort of function it would be greatly appreciated! if I could also do this without listing lots of overloads that would be nice, the real definitions also have a few "keyword only" arguments that would be nice not to have to repeat each time

Answer 1

The reason why you're getting the "Overloaded function implementation does not accept all possible arguments..." error is because your overload implementation does not correctly handle calls that look like this: foo(my_callable, a=3, b=4).

After all, according to your overload signatures, the user could in theory explicitly use named arguments for a, b, c, and so forth -- and so, your implementation needs to support those kinds of calls.

There are two different ways you can fix this.

The first way is to tack on a **kwargs: Any and modify your overload implementation to look like this:

def foo(fn: Callable[..., R], *args: Any, **kwargs: Any) -> Any:
    return fn(*args, **kwargs)

Now your implementation will correctly handle these kinds of calls.

The second way is to prefix each of your parameters with two underscores, like so:

@overload
def foo(fn: Callable[[A], R], __a: A) -> R: ...
@overload
def foo(fn: Callable[[A, B], R], __a: A, __b: B) -> R: ...
@overload
def foo(fn: Callable[[A, B, C], R], __a: A, __b: B, __c: C) -> R: ...

def foo(fn: Callable[..., R], *args: Any) -> Any:
    return fn(*args)

When mypy sees a parameter starting with two underscores, it understands that argument is meant to be positional-only. So, mypy will reject calls like foo(my_fn, __a=3, __b=4).

This is a typing-only thing though. Prefixing your parameters with two underscores has no special meaning at runtime.

---

Regarding your broader question about not having to repeat so many overloads: unfortunately, tacking on a bunch of overloads is the best we can do at the moment. The technique you're using is the same technique typeshed uses to type functions like map(...) and filter(...), for example.

In order to do better, we need a feature called variadic generics -- but they're a complicated feature and mypy unfortunately doesn't support them yet. The plan is to hopefully have them implemented sometime later in 2019 though, so you might be able to rip out the overloads then.