Error loading Python modules when embedding Python in C++ OpenFOAM

Question

TL;DR I'm trying to run Python embedded in OpenFOAM in C++, but including some Python modules is causing OpenFOAM to fail even though the Python script seems to work fine by itself. I've tried appending the module locations to the Python path but it hasn't worked.

I have a long Python script I wrote to couple additional capabilities into an OpenFOAM application. I want to embed this Python script into the OpenFOAM C++ code so that it executes within the iteration loops of the application.

This is what I'm including in OpenFOAM C++ to initialise Python, embed my module and functions, and run it with two arguments (zone name and temperature) passed from OpenFOAM:

// Set PYTHONPATH TO working directory
setenv("PYTHONPATH","../",1); 

Py_Initialize();

PyRun_SimpleString("import sys; sys.path.append('/some/path')\n");

PyObject* myModule = PyImport_ImportModule("couplingScript");

PyObject* myFunction = PyObject_GetAttrString(myModule,"couplingFunction");

PyObject* pZone = PyUnicode_FromString(zonename.c_str());

PyObject* pT = PyFloat_FromDouble(T);

PyObject* pArgs = PyTuple_Pack(2,pZone, pT);

PyObject* myResult = PyObject_CallObject(myFunction, pArgs);

PyErr_Print();

Py_Finalize();

This is a simplified version of my couplingScript.py file, which runs fine from within OpenFOAM:

import sys

def couplingFunction(zone, T):
    print("Zone = " + zone)
    print("Temperature = " + str(T))

couplingFunction('Zone 1', 273)

OUTPUT:

Zone = Zone 1
Temperature = 273

However, I need to include two additional modules - numpy and pandas - to make the full couplingScript file run. This is a simplified example:

import sys
import numpy as np
import pandas as pd

def couplingFunction(zone, T):
    print("Zone = " + zone)
    print("Temperature = " + str(T))
    numpy_check = np.zeros(2)
    pandas_check = pd.DataFrame()
    print(numpy_check)
    print(pandas_check)

couplingFunction('Zone 1', 273)

With these additions, couplingScript.py itself runs fine BUT the embedded Python fails.

 OUTPUT running couplingScript.py from terminal via Python:

 Zone = Zone 1
 Temperature = 273
 Empty DataFrame
 Columns: []
 Index: []
 [0. 0.]

 OUTPUT running couplingScript.py from embedded OpenFOAM application:

 #0  Foam::error::printStack(Foam::Ostream&) at ??:?
 #1  Foam::sigFpe::sigHandler(int) at ??:?
 #2  ? in /lib/x86_64-linux-gnu/libc.so.6
 #3  ? at ??:?
 #4  PyNumber_Multiply in /lib/x86_64-linux-gnu/libpython3.8.so.1.0
 #5  _PyEval_EvalFrameDefault in /lib/x86_64-linux-gnu/libpython3.8.so.1.0
 #6  ? in /lib/x86_64-linux-gnu/libpython3.8.so.1.0
 #7  ? in /lib/x86_64-linux-gnu/libpython3.8.so.1.0
 #8  _PyEval_EvalFrameDefault in /lib/x86_64-linux-gnu/libpython3.8.so.1.0
 #9  _PyEval_EvalCodeWithName in /lib/x86_64-linux-gnu/libpython3.8.so.1.0
 #10  PyEval_EvalCodeEx in /lib/x86_64-linux-gnu/libpython3.8.so.1.0
 #11  PyEval_EvalCode in /lib/x86_64-linux-gnu/libpython3.8.so.1.0
 #12  ? in /lib/x86_64-linux-gnu/libpython3.8.so.1.0
 #13  ? in /lib/x86_64-linux-gnu/libpython3.8.so.1.0
 #14  PyVectorcall_Call in /lib/x86_64-linux-gnu/libpython3.8.so.1.0
 #15  _PyEval_EvalFrameDefault in /lib/x86_64-linux-gnu/libpython3.8.so.1.0
 #16  _PyEval_EvalCodeWithName in /lib/x86_64-linux-gnu/libpython3.8.so.1.0
 #17  _PyFunction_Vectorcall in /lib/x86_64-linux-gnu/libpython3.8.so.1.0
 #18  ? in /lib/x86_64-linux-gnu/libpython3.8.so.1.0
 #19  _PyEval_EvalFrameDefault in /lib/x86_64-linux-gnu/libpython3.8.so.1.0
 #20  ? in /lib/x86_64-linux-gnu/libpython3.8.so.1.0
 #21  ? in /lib/x86_64-linux-gnu/libpython3.8.so.1.0
 #22  _PyEval_EvalFrameDefault in /lib/x86_64-linux-gnu/libpython3.8.so.1.0
 #23  ? in /lib/x86_64-linux-gnu/libpython3.8.so.1.0
 #24  ? in /lib/x86_64-linux-gnu/libpython3.8.so.1.0
 #25  _PyEval_EvalFrameDefault in /lib/x86_64-linux-gnu/libpython3.8.so.1.0
 #26  ? in /lib/x86_64-linux-gnu/libpython3.8.so.1.0
 #27  ? in /lib/x86_64-linux-gnu/libpython3.8.so.1.0
 #28  _PyEval_EvalFrameDefault in /lib/x86_64-linux-gnu/libpython3.8.so.1.0
 #29  ? in /lib/x86_64-linux-gnu/libpython3.8.so.1.0
 #30  ? in /lib/x86_64-linux-gnu/libpython3.8.so.1.0
 #31  _PyObject_CallMethodIdObjArgs in /lib/x86_64-linux-gnu/libpython3.8.so.1.0
 #32  PyImport_ImportModuleLevelObject in /lib/x86_64-linux-gnu/libpython3.8.so.1.0
 #33  ? in /lib/x86_64-linux-gnu/libpython3.8.so.1.0
 #34  ? in /lib/x86_64-linux-gnu/libpython3.8.so.1.0
 #35  _PyObject_MakeTpCall in /lib/x86_64-linux-gnu/libpython3.8.so.1.0
 #36  ? in /lib/x86_64-linux-gnu/libpython3.8.so.1.0
 #37  _PyEval_EvalFrameDefault in /lib/x86_64-linux-gnu/libpython3.8.so.1.0
 #38  _PyEval_EvalCodeWithName in /lib/x86_64-linux-gnu/libpython3.8.so.1.0
 #39  PyEval_EvalCodeEx in /lib/x86_64-linux-gnu/libpython3.8.so.1.0
 #40  PyEval_EvalCode in /lib/x86_64-linux-gnu/libpython3.8.so.1.0
 #41  ? in /lib/x86_64-linux-gnu/libpython3.8.so.1.0
 #42  ? in /lib/x86_64-linux-gnu/libpython3.8.so.1.0
 #43  PyVectorcall_Call in /lib/x86_64-linux-gnu/libpython3.8.so.1.0
 #44  _PyEval_EvalFrameDefault in /lib/x86_64-linux-gnu/libpython3.8.so.1.0
 #45  _PyEval_EvalCodeWithName in /lib/x86_64-linux-gnu/libpython3.8.so.1.0
 #46  _PyFunction_Vectorcall in /lib/x86_64-linux-gnu/libpython3.8.so.1.0
 #47  ? in /lib/x86_64-linux-gnu/libpython3.8.so.1.0
 #48  _PyEval_EvalFrameDefault in /lib/x86_64-linux-gnu/libpython3.8.so.1.0
 #49  ? in /lib/x86_64-linux-gnu/libpython3.8.so.1.0
 #50  ? in /lib/x86_64-linux-gnu/libpython3.8.so.1.0
 #51  _PyEval_EvalFrameDefault in /lib/x86_64-linux-gnu/libpython3.8.so.1.0
 #52  ? in /lib/x86_64-linux-gnu/libpython3.8.so.1.0
 #53  ? in /lib/x86_64-linux-gnu/libpython3.8.so.1.0
 #54  _PyEval_EvalFrameDefault in /lib/x86_64-linux-gnu/libpython3.8.so.1.0
 #55  ? in /lib/x86_64-linux-gnu/libpython3.8.so.1.0
 #56  ? in /lib/x86_64-linux-gnu/libpython3.8.so.1.0
 #57  _PyEval_EvalFrameDefault in /lib/x86_64-linux-gnu/libpython3.8.so.1.0
 #58  ? in /lib/x86_64-linux-gnu/libpython3.8.so.1.0
 #59  ? in /lib/x86_64-linux-gnu/libpython3.8.so.1.0
 #60  _PyObject_CallMethodIdObjArgs in /lib/x86_64-linux-gnu/libpython3.8.so.1.0
 #61  PyImport_ImportModuleLevelObject in /lib/x86_64-linux-gnu/libpython3.8.so.1.0
 #62  _PyEval_EvalFrameDefault in /lib/x86_64-linux-gnu/libpython3.8.so.1.0
 #63  _PyEval_EvalCodeWithName in /lib/x86_64-linux-gnu/libpython3.8.so.1.0
 #64  PyEval_EvalCodeEx in /lib/x86_64-linux-gnu/libpython3.8.so.1.0
 #65  PyEval_EvalCode in /lib/x86_64-linux-gnu/libpython3.8.so.1.0
 #66  ? in /lib/x86_64-linux-gnu/libpython3.8.so.1.0
 #67  ? in /lib/x86_64-linux-gnu/libpython3.8.so.1.0
 #68  PyVectorcall_Call in /lib/x86_64-linux-gnu/libpython3.8.so.1.0
 #69  _PyEval_EvalFrameDefault in /lib/x86_64-linux-gnu/libpython3.8.so.1.0
 #70  _PyEval_EvalCodeWithName in /lib/x86_64-linux-gnu/libpython3.8.so.1.0
 #71  _PyFunction_Vectorcall in /lib/x86_64-linux-gnu/libpython3.8.so.1.0
 #72  ? in /lib/x86_64-linux-gnu/libpython3.8.so.1.0
 #73  _PyEval_EvalFrameDefault in /lib/x86_64-linux-gnu/libpython3.8.so.1.0
 #74  ? in /lib/x86_64-linux-gnu/libpython3.8.so.1.0
 #75  ? in /lib/x86_64-linux-gnu/libpython3.8.so.1.0
 #76  _PyEval_EvalFrameDefault in /lib/x86_64-linux-gnu/libpython3.8.so.1.0
 #77  ? in /lib/x86_64-linux-gnu/libpython3.8.so.1.0
 #78  ? in /lib/x86_64-linux-gnu/libpython3.8.so.1.0
 #79  _PyEval_EvalFrameDefault in /lib/x86_64-linux-gnu/libpython3.8.so.1.0
 #80  ? in /lib/x86_64-linux-gnu/libpython3.8.so.1.0
 #81  ? in /lib/x86_64-linux-gnu/libpython3.8.so.1.0
 #82  _PyEval_EvalFrameDefault in /lib/x86_64-linux-gnu/libpython3.8.so.1.0
 #83  ? in /lib/x86_64-linux-gnu/libpython3.8.so.1.0
 #84  ? in /lib/x86_64-linux-gnu/libpython3.8.so.1.0
 #85  _PyObject_CallMethodIdObjArgs in /lib/x86_64-linux-gnu/libpython3.8.so.1.0
 #86  PyImport_ImportModuleLevelObject in /lib/x86_64-linux-gnu/libpython3.8.so.1.0
 #87  ? in /lib/x86_64-linux-gnu/libpython3.8.so.1.0
 #88  ? in /lib/x86_64-linux-gnu/libpython3.8.so.1.0
 #89  _PyObject_MakeTpCall in /lib/x86_64-linux-gnu/libpython3.8.so.1.0
 #90  ? in /lib/x86_64-linux-gnu/libpython3.8.so.1.0
 #91  PyObject_CallFunction in /lib/x86_64-linux-gnu/libpython3.8.so.1.0
 #92  PyImport_Import in /lib/x86_64-linux-gnu/libpython3.8.so.1.0
 #93  PyImport_ImportModule in /lib/x86_64-linux-gnu/libpython3.8.so.1.0
 #94  ? in ~/OpenFOAM/ud215-v1906/platforms/linux64GccDPInt32Opt/bin/GeN-Foam
 #95  ? in ~/OpenFOAM/ud215-v1906/platforms/linux64GccDPInt32Opt/bin/GeN-Foam
 #96  ? in ~/OpenFOAM/ud215-v1906/platforms/linux64GccDPInt32Opt/bin/GeN-Foam
 #97  __libc_start_main in /lib/x86_64-linux-gnu/libc.so.6
 #98  ? in ~/OpenFOAM/ud215-v1906/platforms/linux64GccDPInt32Opt/bin/GeN-Foam
 Floating point exception (core dumped)

Things I have tried:

• Installing numpy/pandas via pip gives me:

  • Requirement already satisfied: numpy in /usr/lib/python3/dist-packages (1.17.4)
  • Requirement already satisfied: pandas in /usr/lib/python3/dist-packages (0.25.3)

• Appending this to the python path via sys.path.append('/usr/lib/python3/dist-packages') directly after import sys in couplingScript.py:

  • This gives me the same error as before when run via OpenFOAM

• Importing directly from the package location using:

  • import usr.lib.python3.dist-packages.numpy as np
  • import usr.lib.python3.dist-packages.pandas as pd

• I also tried:

  • from .usr.lib.python3.dist-packages import numpy as np
  • from .usr.lib.python3.dist-packages import pandas as pd

These last two give a slightly shorter error:

 #0  Foam::error::printStack(Foam::Ostream&) at ??:?
 #1  Foam::sigSegv::sigHandler(int) at ??:?
 #2  ? in /lib/x86_64-linux-gnu/libc.so.6
 #3  PyObject_GetAttrString in /lib/x86_64-linux-gnu/libpython3.8.so.1.0
 #4  ? in ~/OpenFOAM/ud215-v1906/platforms/linux64GccDPInt32Opt/bin/GeN-Foam
 #5  ? in ~/OpenFOAM/ud215-v1906/platforms/linux64GccDPInt32Opt/bin/GeN-Foam
 #6  ? in ~/OpenFOAM/ud215-v1906/platforms/linux64GccDPInt32Opt/bin/GeN-Foam
 #7  __libc_start_main in /lib/x86_64-linux-gnu/libc.so.6
 #8  ? in ~/OpenFOAM/ud215-v1906/platforms/linux64GccDPInt32Opt/bin/GeN-Foam
 Segmentation fault (core dumped)`

I have also run print(sys.path) for both runs, which gives:

Python run from terminal:

 ['/home/abc123/Coupled_model/GF', '/usr/lib/python38.zip', '/usr/lib/python3.8', '/usr/lib/python3.8/lib-dynload', '/home/abc123/.local/lib/python3.8/site-packages', '/usr/local/lib/python3.8/dist-packages', '/usr/lib/python3/dist-packages']

Embedded python in OpenFOAM:

 ['/home/abc123/Coupled_model/GF', '/usr/lib/python38.zip', '/usr/lib/python3.8', '/usr/lib/python3.8/lib-dynload', '/home/abc123/.local/lib/python3.8/site-packages', '/usr/local/lib/python3.8/dist-packages', '/usr/lib/python3/dist-packages']

If they're the same, why isn't the embedded python recognising numpy and pandas???

Answer 1

Try deleting and reinstalling the packages.

Answer 2

My guess is that the import of the libraries is not seen as they are called before the definition of "couplingFunction"

PyRun_SimpleString("import sys; sys.path.append('/some/path')\n");

PyObject* myModule = PyImport_ImportModule("couplingScript");

PyObject* myFunction = PyObject_GetAttrString(myModule,"couplingFunction");

Have you tried the following definition?

def couplingFunction(zone, T):
    import numpy as np 
    import pandas as pd
    print("Zone = " + zone)
    print("Temperature = " + str(T))
    numpy_check = np.zeros(2)
    pandas_check = pd.DataFrame()
    print(numpy_check)
    print(pandas_check)

Alternatively, perhaps you could try:

PyRun_SimpleString("import sys; sys.path.append('/some/path'); import numpy as np; import pandas as pd;\n");