I'm doing some scientific computation in python, and recently switched away from Anaconda since it doesn't yet support python 3.10 (which has some new features I'd like to use). I am now running on python version 3.10.9. I have a pipenv virtual environment set up for this, and in it I've got numpy version 1.23.5 installed. (Trying to upgrade to a newer version fails with a bunch of errors -- if upgrading numpy ends up being necessary I'll need to open a new question to ask about how to get that working.)
Everything initially seems to be working fine, until I try to run a parallelized numpy computation. At that point, I get the error DLL load failed while importing _multiarray_umath: The specified module could not be found. Here's the associated traceback:
Traceback (most recent call last):
File "C:\Users\charles\.virtualenvs\pipenv-WB8juvy1\lib\site-packages\numpy\core\__init__.py", line 23, in <module>
from . import multiarray
File "C:\Users\charles\.virtualenvs\pipenv-WB8juvy1\lib\site-packages\numpy\core\multiarray.py", line 10, in <module>
from . import overrides
File "C:\Users\charles\.virtualenvs\pipenv-WB8juvy1\lib\site-packages\numpy\core\overrides.py", line 6, in <module>
from numpy.core._multiarray_umath import (
ImportError: DLL load failed while importing _multiarray_umath: The specified module could not be found.
I can import numpy just fine, and I can use various functions from numpy without issue as well. I only get problems when I specifically want to do some parallelized computations. At that point my console gets flooded with this error (as far as I can tell, it's printing nearly exactly the same thing over and over, presumably once for every worker that it tried to launch).
The following text is also included in the error message:
IMPORTANT: PLEASE READ THIS FOR ADVICE ON HOW TO SOLVE THIS ISSUE!
Importing the numpy C-extensions failed. This error can happen for
many reasons, often due to issues with your setup or how NumPy was
installed.
We have compiled some common reasons and troubleshooting tips at:
https://numpy.org/devdocs/user/troubleshooting-importerror.html
Please note and check the following:
* The Python version is: Python3.10 from "C:\Users\charles\.virtualenvs\pipenv-WB8juvy1\Scripts\python.exe"
* The NumPy version is: "1.24.1"
and make sure that they are the versions you expect.
Please carefully study the documentation linked above for further help.
While this looks pretty promising, I've read through the documentation on that page and nothing seems to be addressing the particular issue I'm running into here.
Additional details about my setup: The virtual environment was set up with pipenv, using the --site-packages flag (as I don't seem to be able to properly install PyQt5 in the virtual environment, so I'm just sourcing it from the system). I've got numpy installed in both my system's site-packages - that is version 1.24.1 (which checks out with the error messages) - and in the virtual environment's site packages (I didn't explicitly install it there, it was installed as a dependency; the virtual version is 1.23.5). Uninstalling numpy from the system site packages does not fix the issue.
I've also tried completely deleting the virtual environment and reinstalling it from the pipfile, but that also doesn't fix the issue. The version of numpy which is installed in the virtual environment does indeed have a DLL in its .lib folder, so it's not just missing. As far as I can tell there's nothing weird about my path; just in case I tried manually adding the .lib folders where the DLLs live to the path. I added both the virtual and system site packages to the path, and tried sourcing them in various different orders, to no avail.
I'm kinda at the end of my ability to troubleshoot this at this point, hence this request for help. In the meantime I'm resorting to doing all of my computations serially, but (as you might imagine) this is very slow and not very sustainable for the long term. If nobody can help me fix this, I'm going to have to just go back to Anaconda and give up on features from the newer python versions until they support them. (Or maybe I can try working without a virtual environment, but I've messed up systems pretty badly that way before and I'm kind of loathe to take that approach).
