1

I have a program written in Fortran and compiled (with -fPIC) as a dynamic library. I load with CDLL in Python to perform some numerical calculations. The library uses internal persistent memory (in modules) to save intermediate values. I work both on Windows and Linux (with a .dll and an .so, respectively). The dynamic library is packaged inside the Python module and installed in a lib subdirectory.

If I run two different Python scripts that load the dynamic library on the same computer, would they get a separate internal memory or will they access the same one (thus, corrupting the results)?

How would this be affected if I:

  • Load them inside a different multiprocessing instance
  • Load them inside a different multithreading instance
  • Load them in two different Jupyter notebooks with different kernel
  • Run them in different Python virtual environments

There is a similar question. However, since the dynamic library is included in the Python module, I want to avoid having to rename and move around files.

electrique
  • 431
  • 6
  • 18

1 Answers1

2

If I run two different Python scripts that load the dynamic library on the same computer, would they get a separate internal memory or will they access the same one (thus, corrupting the results)?

  • The same library can only be loaded once; more precisely as long as you try to load a library with the same path it gets only loaded once in the process.

    • Now supposing that you load a library, copy / paste that library elsewhere, rename it and try to load the copy then it gets loaded twice.

  • As for the memory used by the same library loaded multiple times, as you get only one library loaded, there's no corruption. If you load a copy of it, then each of those library gets its own memory allocations.

Load them inside a different multiprocessing instance

No problems. multiprocessing uses different processes and different processes can't overwrite anything into another process address space (unless explicitly asked for it).

Load them inside a different multithreading instance

You'll only be able to load one instance of the library. That being said as you only have one instance of the library, technically and depending on the library, a call to a function might overwrite the previous internal state of the library. It heavily depends if the library can be used in a multi-threading context.

As you said the library keeps an internal state, each call to a function has a chance to overwrite / change the previous internal state (depending on the effects of the called function on the internal state).

Load them in two different Jupyter notebooks with different kernel

Depends if the two notebooks are using two different instance of the interpreter or not.

  • If it's the same interpreter running two notebooks, then you only have one instance of your library loaded by the interpreter.

  • Different interpreters: two processes -> two libraries with each its own separated state.

Run them in different Python virtual environments

Different virtual environments uses different processes.

Neitsa
  • 7,693
  • 1
  • 28
  • 45