3

This question has been bothering me for two weeks and I've searched online and asked people but couldn't get an answer.

Python by default build the library libpythonMAJOR.MINOR.a and statically links it into the interpreter. Also it has an --enable-shared flag, which will build a share library libpythonMAJOR.MINOR.so.1.0, and dynamically link it to the interpreter.

Based on my poor CS knowledge, the first thought came into my mind when I saw "shared library", is that, "the shared bulid one must save a lot of memory compared to the static build one!".

Then I had this assumption:

# share build
34K Jun 29 11:32 python3.9
21M Jun 29 11:32 libpython3.9.so.1.0

10 shared python processes, mem usage = 0.034M * 10 + 21M ≈ 21M

# static build
22M Jun 27 23:45 python3.9

10 static python processes, mem usage = 10*22M = 220M

shared python wins!

Later I ran a toy test on my machine and found that's wrong.

test.py

import time
i = 0
while i < 20:
        time.sleep(1)
        i += 1

print('done')

mem_test.sh

#! /bin/bash
for i in {1..1000}
do 
         ./python3.9 test.py &
done

For share python to run I set export LD_LIBRARY_PATH=/home/tian/py3.9.13_share/lib .

I ran mem_test.sh separately (one by one) with 2 pythons and simply monitored the total mem usage via htop in another console. It turns out that both eat almost the same amount of memory.

Later on people taught me there's something call "paging on demand":

Is an entire static program loaded into memory when launched?

How does an executable get loaded into RAM, does the whole file get loaded into RAM even when the whole file won't be needed, or does it get loaded in "chunks"?

so my previous calculation of static python mem usage is completely wrong.

Now I am confused. Shared build python doesn't use less memory via a share library runtime?

Question:

What's the benefit of a shared build python vs a static build python? Or the shared built python indeed save some memory by the mechanism of using a share library, but my test is too trival to reveal?

P.S.

Checking some python official Dockerfiles, e.g. this one you would see they all set --enable-shared.

Also there's related issue on pyenv https://github.com/pyenv/pyenv/issues/2294 , it seems that neither they figure that out.

Rick
  • 7,007
  • 2
  • 49
  • 79
  • Shared vs static is more about their disk storage needs. Dynamic linking uses shared libraries and, because they're shared, there only needs to be one instance on disk. Statically linked libraries are part of the executable and, if another application also needs that library, there would be more than one copy of it on disk. – Ouroborus Jul 11 '22 at 18:27
  • Sharing saves *disk space*, as you aren't including the library code in multiple executable files. Operating systems don't tend to allow processes to share memory, even read-only memory. – chepner Jul 11 '22 at 18:27
  • I believe there is a growing trend towards static linking, as disk space is not nearly at the same premium as it was 25-30 years ago. Docker has a different set of priorities, attempting to keep images small to decrease network bandwidth usage. (Also keeping shared libraries in a deeper layer could speed up image rebuild time if all you need to rebuild is the executable, not its dependencies.) – chepner Jul 11 '22 at 18:30
  • @Ouroborus *disk space*? Such a simple and trivial reason? Per your comment, *"Dynamic linking uses shared libraries and, because they're shared, there only needs to be one instance on disk"*, different version python builds has different `libpythonMAJOR.MINOR.so.1.0`, and used by the corresponding python executable solely. I don't see how that saves space. The system wide python share library might do what you say. – Rick Jul 11 '22 at 18:40
  • That's not the only .so that's linked. I'm not sure if you can choose dynamic vs static linking on a per-object basis, I've always seen the choice made as an all or nothing kind of thing. On Windows, these are usually DLLs (`pythonMAJORMINOR.dll`, I think, in this case) and there's at least two executables that use them: `python.exe` and `pythonw.exe`. – Ouroborus Jul 11 '22 at 19:17
  • If you just run python, you don't need enable shared. However, you won't be able to import any python module that expects it. For example, if you build a c++ python extention. – Kenny Ostrom Jul 11 '22 at 20:27
  • @KennyOstrom What do you mean? It doesn’t have to be a share build python to import a C extension. For static build python, all requied symbols are built into the interpreter executable. – Rick Jul 12 '22 at 00:30

2 Answers2

0

Apart from disk usage purpose as mentioned by @Ouroborus, I think there's also the 'convenience of updating' purpose: suppose you have installed a version of python that turned out to have a critical security problem. Then, all software using python might be exposed. To fix the problem, you need to update all the pythons in your computer. If this is a shared python, you only need to update a few files, but if a software uses a statically built python, then you have to update that entire software to get an update on python.

However, this benefit may also be regarded as a downside: updating python may introduce breaking changes, and certain software may break, and during the update process, we are unable to detect these breakages.

ice1000
  • 6,406
  • 4
  • 39
  • 85
0

It turns out to be that others are talking about the scenario "Embedding Python in Another Application" (https://docs.python.org/3/extending/embedding.html).

If that's the case, then "saving disk space" and other mentioned reasons make sense. Because embedding python in another application, either you need to statically link libpythonMAJOR.MINOR.a or dynamically link libpythonMAJOR.MINOR.so.1.0.

So my current conclusion is that whether python is shared built or statically built only affects the "Embedding Python in Another Application" scenario. For normal use cases, e.g. running the python interpeter, it doesn't make much diffferences.

Update:

Disk usage comparsion, see comments in makfile:

https://stackoverflow.com/a/73099136/5983841

Rick
  • 7,007
  • 2
  • 49
  • 79