Why does adding multiprocessing prevent python from finding my compiled c program?

Question

I am currently looking to speed up my code using the power of multiprocessing. However I am encountering some issues when it comes to calling the compiled code from python, as it seems that the compiled file disappears from the code's view when it includes any form of multiprocessing.

For instance, with the following test code:

#include <omp.h>

int main() {
    int thread_id;
#pragma omp parallel
    {
        thread_id = omp_get_thread_num();
    }
    return 0;
}

Here, I compile the program, then turn it into a .so file using the command

gcc -fopenmp -o theories/test.so -shared -fPIC -O2 test.c

I then attempt to run the said code from test.py:

from ctypes import CDLL
import os

absolute_path = os.path.dirname(os.path.abspath(__file__))
# imports the c libraries
test_lib_path = absolute_path + '/theories/test.so'
test = CDLL(test_lib_path)
test.main()
print('complete')

I get the following error:

FileNotFoundError: Could not find module 'C:\[my path]\theories\test.so' (or one of its dependencies). Try using the full path with constructor syntax.

However, when I comment out the multiprocessing element to get the follwing code:

#include <omp.h>

int main() {
    int thread_id;
    /*
#pragma omp parallel
    {
        thread_id = omp_get_thread_num();
    }
    */
    return 0;
}

I then have a perfect execution with the python program printing out "complete" at the end.

I'm wondering how this has come to happen, and how the code can seemingly be compiled fine but then throw problems only once it's called from python (also I have checked and the file is in fact created).

UPDATES:

1. I have now checked that I have libgomp-1.dll installed
2. I have uninstalled and reinstalled MinGW, with no change happening.
3. I have installed a different, 64 bit version of gcc and, using a different (64 bit python 3.10) version of python have reproduced the same error. This also has libgomp-1.dll.

Answer 1

Note where the error message says "or one of its dependencies".

Try running ldd on your test.so file to see if it's completely linked.

EDIT1:

Generally, gcc requires binutils. (On ms-windows, I guess they could be combined.) Which means that you should have objdump.

If you run objdump -x test.so|more, you should see some lines starting with "NEEDED" in the "Dynamic section". Those are the shared libraries needed by this one.

Answer 2

I think this has the same root cause as [SO]: Can't import dll module in Python (@CristiFati's answer) (also check [SO]: PyWin32 and Python 3.8.0 (@CristiFati's answer)).

A .dll (.so) is only loaded when its dependencies are successfully loaded (recursively).
[SO]: Python Ctypes - loading dll throws OSError: [WinError 193] %1 is not a valid Win32 application (@CristiFati's answer) focuses on a different error, but covers this topic.

When commenting out omp_get_thread_num call, the linker no longer links test.so to libgomp*.dll (as it doesn't need anything from it), and code runs fine (all required .dlls are found).

To fix the issue, you should add to os.add_dll_directory (before attempting to load the .so):

1. libgomp*.dll's directory

2. MinGW's bin directory

3. Any other directory that may contain .dlls that are required (dependents)

To see a .dll dependencies, check [SO]: Discover missing module using command-line ("DLL load failed" error) (@CristiFati's answer).

Notes:

• main as an exported function from an so can be misleading

• Although not required here (at least for the current code), check [SO]: C function called from Python via ctypes returns incorrect value (@CristiFati's answer)