56

I am trying to understand the lower level implementations of python 3. There is one module named _posixsubprocess used by the subprocess module. I tried to find the location of this module in my system and found that it's a stub file.

Could someone guide me as I have no idea about what are the stub files and how are they implemented at the lower level?

keineahnung2345
  • 2,635
  • 4
  • 13
  • 28
Rahul Gusai
  • 701
  • 1
  • 6
  • 11
  • Maybe you were looking for this? [Modules/_posixsubprocess.c](https://github.com/python/cpython/blob/f3751efb5c8b53b37efbbf75d9422c1d11c01646/Modules/_posixsubprocess.c) – exhuma Nov 27 '19 at 09:23
  • So it is essentially a Cython module that can directly be integrated into python interpreter being a C compiled code, correct me if I am wrong? – Rahul Gusai Nov 27 '19 at 11:05
  • "directly integrated" is maybe not the wording I would choose. When compiled it will become a `.so`/`.pyd` file which can be imported if it's findable on the path. You can "easily" create your own modules in C. I doubt that it's `cython` though. It's more likely that it uses [the standard way of writing C extensions](https://docs.python.org/3/extending/building.html) – exhuma Nov 27 '19 at 12:24
  • btw: You mention `_posixsubprocess`, which is - in CPython - a `.c` file, But you also mention that you see it as a `.pyi` file. Where exactly did you find that `.pyi` file? Do you have a link to the Python source repository? – exhuma Nov 27 '19 at 12:28
  • .. nevermind... I found it in the "typeshed". I will update my answer – exhuma Nov 27 '19 at 12:29
  • I think the answer should be a bit clearer now. – exhuma Nov 27 '19 at 12:34
  • 1
    I was confusing the standard way of writing C extensions for python with `python`. It's cleared now. Also, I found this link which will be quite helpful to understand implementation level details https://realpython.com/cpython-source-code-guide/#part-1-introduction-to-cpython – Rahul Gusai Nov 27 '19 at 19:34

1 Answers1

88

_posixsubprocess

The file you are referencing is a Python module written in C. It's not a "stub" file. The real implementation can be found in the stdlib at Modules/_posixsubprocess.c. You can see how writing a C/C++ extension is written by having a look at Building C and C++ Extensions. This should help you understanding the code in _posixsubprocess.c.

In order to add type-hints to that file (which is an "Extension Module" as it is written in C), the type hints are added to a "stub" file with the extension .pyi.

That file can be found in the typeshed which is a collection of stub files. The typeshed also contains stubs for third-party modules which is a historical remnant. That is no longer needed since PEP-561 has been adopted.

Concerning stub/pyi files

Stub files contain type-hinting information of normal Python modules. The full official documentation can be found in the section about stub-files in PEP-484.

For example, if you have a Python module mymodule.py like this:

def myfunction(name):
   return "Hello " + name

Then you can add type-hints via a stub-file mymodule.pyi. Note that here the ellipsis (...) is part of the syntax, so the code-block below really shows the complete file contents:

def myfunction(name: str) -> str: ...

They look very similar to C header files in that they contain only the function signatures, but their use is purely optional.

You can also add type hints directly in the .py module like the following:

def myfunction(name: str) -> str:
   return "Hello " + name

But there are some cases where you want to keep them separate in stubs:

  • You want to keep your code Python 2 compatible and don't like the # type: ... comment syntax
  • You use function annotations for something else but still want to use type-hints
  • You are adding type-hints into an existing code-base and want to keep code-churn in existing files minimal
exhuma
  • 20,071
  • 12
  • 90
  • 123