Python introspection: get the argument list of a method_descriptor?

Question

A code illustration as an intro to my questions:

import re, inspect, datetime

inspect.getargspec (re.findall)
# =>
# ArgSpec(args = ['pattern', 'string', 'flags'], varargs=None,
# keywords=None, defaults = (0,))

type (datetime.datetime.replace)
# => <type 'method_descriptor'>

inspect.getargspec (datetime.datetime.replace)
# => Traceback (most recent call last):
#      File "<stdin>", line 1, in <module>
#      File "/usr/lib/python2.7/inspect.py", line 816, in getargspec
#        raise TypeError('{!r} is not a Python function'.format(func))
# TypeError: <method 'replace' of 'datetime.datetime' objects> is
# not a Python function

It seems that the only way for me to find the signature of datetime.datetime.replace while I code is to look it up in the doc: date.replace(year, month, day).

The only introspection part that seems to work:

datetime.datetime.replace.__doc__
# => 'Return datetime with new specified fields.'

I've examined how the Jupyter function arglist tool-tip works, they have the exact same problem, i.e. no arglist available for datetime.datetime.replace.

So here are the questions:

1. Is it still possible to get the argument list somehow? Maybe I could install the C sources for datetime and connect them via the __file__ attribute?

2. Is it possible to annotate a <type 'method_descriptor'> with the arglist information? In that case, I could parse the linked doc's markdown definition and automatically annotate the built-in module functions.

Answer 1

No, you can't get more information; installing the C sources would not give you easy access to the same. That's because most methods defined in C code do not actually expose this information; you'd have to parse out a rather cryptic piece of C code:

if (! PyArg_ParseTupleAndKeywords(args, kw, "|iiiiiiiO$i:replace",
                                  datetime_kws,
                                  &y, &m, &d, &hh, &mm, &ss, &us,
                                  &tzinfo, &fold))

The re.findall() function is a pure Python function, so is introspectable.

I said most methods defined in C, because as of Python 3.4 and up, methods that use the new Argument Clinic preprocessor will include a new __text_signature__ attribute, which the internal inspect._signature_fromstr() function can parse. This means that even for such C-defined methods, you can introspect the arguments:

>>> import io
>>> import inspect
>>> type(io.BytesIO.read)
<class 'method_descriptor'>
>>> inspect.signature(io.BytesIO.read)
<Signature (self, size=None, /)>

Also see What are __signature__ and __text_signature__ used for in Python 3.4

The datetime module has not yet received much Argument Clinic love. We'll have to be patient, or if you really care a lot about this, supply patches that convert the module to using Argument Clinic.

If you want to see what modules do have support already, look at the Modules/clinic subdirectory which contains the generated clinic output; for the datetime module, only datetime.datetime.now() is currently included. That method defines a clinic block:

/*[clinic input]
@classmethod
datetime.datetime.now
    tz: object = None
        Timezone object.
Returns new datetime object representing current time local to tz.
If no tz is specified, uses local timezone.
[clinic start generated code]*/

static PyObject *
datetime_datetime_now_impl(PyTypeObject *type, PyObject *tz)
/*[clinic end generated code: output=b3386e5345e2b47a input=80d09869c5267d00]*/

making the method introspectable:

>>> import datetime
>>> inspect.signature(datetime.datetime.now)
<Signature (tz=None)>

There is no way to directly attach information to those C functions and methods that are not introspectable; they don't support attributes either.

Most autocomplete solutions that want to support such objects use separate data structures where the information is maintained independently (with all the inherent risks of the data getting out of sync). Some of these are available for your own purposes:

• The Komodo IDE code intelligence library (open source, used other editors too) uses the CIX format to encode this data; you could download the Python 3 catalog. Unfortunately for your specific example, the datetime.replace() function signature has not been fleshed out either:

  <scope doc="Return datetime with new specified fields." ilk="function" name="replace" />
  

• The new Python 3.5 type hinting syntax also needs to know what types of arguments objects expect, and to this end stub files need to be provided for objects that can't be introspected. The Python typeshed project provides these. This includes all argument names for the datetime module:

  class datetime:
      # ...
      def replace(self, year: int = ..., month: int = ..., day: int = ..., hour: int = ...,
          minute: int = ..., second: int = ..., microsecond: int = ..., tzinfo:
          Optional[_tzinfo] = None) -> datetime: ...
  

  You'd have to parse such a file yourself; they can't always be imported as the stubs reference types not yet defined, rather than use forward references:

  >>> import importlib.machinery
  >>> path = 'stdlib/3/datetime.pyi'
  >>> loader = importlib.machinery.SourceFileLoader('datetime', path)
  >>> loader.load_module()
  Traceback (most recent call last):
    File "<stdin>", line 1, in <module>
    File "<frozen importlib._bootstrap_external>", line 399, in _check_name_wrapper
    File "<frozen importlib._bootstrap_external>", line 823, in load_module
    File "<frozen importlib._bootstrap_external>", line 682, in load_module
    File "<frozen importlib._bootstrap>", line 251, in _load_module_shim
    File "<frozen importlib._bootstrap>", line 675, in _load
    File "<frozen importlib._bootstrap>", line 655, in _load_unlocked
    File "<frozen importlib._bootstrap_external>", line 678, in exec_module
    File "<frozen importlib._bootstrap>", line 205, in _call_with_frames_removed
    File "stdlib/3/datetime.pyi", line 12, in <module>
      class tzinfo:
    File "stdlib/3/datetime.pyi", line 13, in tzinfo
      def tzname(self, dt: Optional[datetime]) -> str: ...
  NameError: name 'datetime' is not defined
  

  You may be able to work around that by using a pre-defined module object and globals, then iterating on name errors until it imports though. I'll leave that as an exercise for the reader. Mypy and other type checkers don't try to execute the code, they merely build an AST.

Answer 2

The issue you are having is caused by the fact that C-coded functions do not expose their signature. You'll find more information on this answer to "How to find out the arity of a method in Python".

In your case, re.findall is defined in Python (see def findall(pattern, string, flags=0):) while datetime.datetime.replace is written in C (see datetime_replace(PyDateTime_DateTime *self, PyObject *args, PyObject *kw)).

You can see see using the different attributes available (and the __code__ attribute in particular) on the function with the dir builtin :

>>> dir(datetime.datetime.replace)
['__call__', '__class__', '__delattr__', '__doc__', '__format__', '__get__', '__getattribute__', '__hash__', '__init__', '__name__', '__new__', '__objclass__', '__reduce__', '__reduce_ex__', '__repr__', '__setattr__', '__sizeof__', '__str__', '__subclasshook__']
>>> dir(re.findall)
['__call__', '__class__', '__closure__', '__code__', '__defaults__', '__delattr__', '__dict__', '__doc__', '__format__', '__get__', '__getattribute__', '__globals__', '__hash__', '__init__', '__module__', '__name__', '__new__', '__reduce__', '__reduce_ex__', '__repr__', '__setattr__', '__sizeof__', '__str__', '__subclasshook__', 'func_closure', 'func_code', 'func_defaults', 'func_dict', 'func_doc', 'func_globals', 'func_name']
>>> datetime.datetime.replace.__code__
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
AttributeError: 'method_descriptor' object has no attribute '__code__'
>>> re.findall.__code__
<code object findall at 0x7fe7234e74b0, file "/usr/lib/python2.7/re.py", line 173>

Usually, the help gives you what you need (based on the __doc__ attribute) but in your case, it doesn't seem to help much:

>>> help(datetime.datetime.replace)
Help on method_descriptor:

replace(...)
    Return datetime with new specified fields.

Also, an idea could be to try to set the __code__ attribute to something corresponding to your needs but you can't tweak much on builtin types without subclassing.