How can I learn more about Python’s internals?

Question

I have been programming using Python for slightly more than half an year now and I am more interested in Python internals rather than using Python to develop applications. Currently I am working on porting a few libraries from Python2 to Python3. However, I have a rather abstract view on how to make port stuff over from Python2 to Python3 as most of the changes deal with design issues in Python2.x

I'd like to learn more about Python internals; should I go for a top-down or a bottom-up approach? Are there any references you could recommend?

Can you explain why you think you need to know anything at all about the internals of Python (2.x? 3.x? both) to be able to port code from 2 to 3? — John Machin, Jul 21 '10 at 11:15
perhaps the question was'nt clear enough(sorry about that!). I am currently not finding any kind of difficulty porting libraries, its just that I'd like to have a better understanding of Python internals. I am just curious on how stuff works under-the-hood. I was wondering if a top-down or a bottom-up approach for the same would be better. Also, if someone could suggest resources for the same. — , Jul 21 '10 at 11:28
@uki extremely late answer but I will just put this here for anyone that may still be interested. [Inside The Python Virtual Machine](https://leanpub.com/insidethepythonvirtualmachine) provides a walk through of the python internals in sufficient details. — cobie, Nov 13 '16 at 16:25

score 42 · Accepted Answer · answered Jul 21 '10 at 11:33

It sounds like you want to know more about the rationale behind the design of the language, rather than internals. "internals" to me means things like how objects are laid out in memory, how reference counting works, and so on.

If you're looking for a deeper understanding of the design decisions, try reading the PEPs: they are the proposals for changes in the language, and often include detailed discussions of the reasons for the changes, rejected alternatives, and so on. Even the rejected PEPs are useful, because they show the thinking that has shaped the language.

For example:

and so on..

If you really want to learn about Python internals, then start by reading about the Python C API, which is used to build Python itself: my talk A Whirlwind Excursion through Python C Extensions is one place to start. Then you can dive into the Python source code itself for anything you need to learn about.

Sticking with Ned's theme, you can get the history behind some of the design decisions straight from the horse's mouth - http://python-history.blogspot.com/ — Jeremy Brown, Jul 21 '10 at 14:22

score 15 · Answer 2 · answered May 09 '11 at 04:17

15

To someone who is stumbling upon this question from related links or search, there is a documentation written Yaniv Aknin on Python Internals. It starts from the scratch and is highly readable.

answered May 09 '11 at 04:17

Senthil Kumaran

54,681
14
94
131

score 5 · Answer 3 · edited May 23 '17 at 11:55

5

I find the series of Yaniv Aknin's Pythons Innards series fantastic, too

I discovered it thanks to Planet Python

.

You may be also interested by the answer of TryPyPy in this SO thread

edited May 23 '17 at 11:55

Community

1
1

answered May 09 '11 at 06:48

eyquem

26,771
7
38
46

3

replace the Pythons Innards link with: https://web.archive.org/web/20120225020122/https://en.wordpress.com/tag/pythons-innards/ – Adorn Dec 20 '17 at 15:31
Maybe this is were Yaniv Aknin's Pythons Innards series have moved: https://tech.blog.aknin.name/category/my-projects/pythons-innards/ – vasili111 Jan 30 '22 at 22:12

score 2 · Answer 4 · answered Jul 21 '10 at 11:22

I would first read the What's New document for Python 3. It gives a good high-level overview and touches on the detailed changes.

You might also do a search for 'porting to python 3' or similar. There are lots of good resources and tools.

One tool that's new and hard to find is six, by Benjamin Peterson. It enables writing of code that is compatible across the Python 2*3 gap.

The part I found most difficult about maintaining Python 2 and Python 3 -compatible code was deployment. I could write code that would run just fine, but when I went do package and deploy, it was unclear when the conversion should happen. I ultimately found a distutils command build_py_2_to_3 that would do the trick. By using that command in my setup.py, I could release a source distribution that would deploy on either Python 2 or Python 3. An example can be found in jaraco.util.

You also asked about the internals. If you really want to get at the internals, you can view the source for Python 2.x and Python 3.x, though honestly, I would stick with reading the tutorials and maybe some of the .py files in the Python libs.

I'm not having any trouble porting libraries as such. I am familiar with _how_ to port libraries. I would like to have some help on what kind of approach to take to understand Python internals. Sorry if the question was unclear. — , Jul 21 '10 at 11:30

score 1 · Answer 5 · answered Jul 21 '10 at 13:09

1

should I go for a top-down or a bottom-up approach?

Both! Seriously.

answered Jul 21 '10 at 13:09

Paul D. Waite

96,640
56
199
270

score 0 · Answer 6 · answered Jul 21 '10 at 11:22

0

Have you tried this?

Automated Python 2 to 3 code translation

answered Jul 21 '10 at 11:22

ssoler

4,884
4
32
33

yes. but 2to3 does not take care of all changes. also, doing the conversion is not my problem; i'd like to understand the changes at a more basic level. – Jul 21 '10 at 11:25
Aw, didn’t really deserve a downvote, it was just a misunderstanding. Voted up to compensate. – Paul D. Waite Jul 21 '10 at 13:10

How can I learn more about Python’s internals?

6 Answers6

Linked