0

System: Windows 10
Python: 3.8
Sympy: 1.8

I have a large Sympy Matrix regressor that I need to convert to C code (to speed up later calculations with this matrix). I am using Sympy's autowrap to do this:

self.cy = autowrap(self.regressor, backend='cython', args=tuple(self._get_state()), verbose=True, tempdir=self.__cython_path)

This has been working for a long time now until I did some changes in the matrix (which likely increased its size). When I run the code now, I get the following error messages and I have no idea how to fix this:

Traceback (most recent call last):
  File "C:\Users\Jonas\anaconda3\envs\robot-parameter-identification\lib\site-packages\sympy\core\cache.py", line 72, in wrapper
    retval = cfunc(*args, **kwargs)
TypeError: unhashable type: 'MutableDenseMatrix'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "C:\Users\Jonas\anaconda3\envs\robot-parameter-identification\lib\site-packages\sympy\utilities\autowrap.py", line 167, in _process_files
    retoutput = check_output(command, stderr=STDOUT)
  File "C:\Users\Jonas\anaconda3\envs\robot-parameter-identification\lib\subprocess.py", line 411, in check_output
    return run(*popenargs, stdout=PIPE, timeout=timeout, check=True,
  File "C:\Users\Jonas\anaconda3\envs\robot-parameter-identification\lib\subprocess.py", line 512, in run
    raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command '['C:\\Users\\Jonas\\anaconda3\\envs\\robot-parameter-identification\\python.exe', 'setup.py', 'build_ext', '--inplace']' returned non-zero exit status 1.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "C:/Users/Jonas/Repo/robot-parameter-identification/optimize_trajectory.py", line 39, in <module>
    robot = dynamics.robot_constructor(r, identity)
  File "C:\Users\Jonas\Repo\robot-parameter-identification\identification\dynamics.py", line 473, in robot_constructor
    robot = Robot(robot_path, cython_path, parameters)
  File "C:\Users\Jonas\Repo\robot-parameter-identification\identification\dynamics.py", line 448, in __init__
    self.__get_regressor_objects(cython_path)
  File "C:\Users\Jonas\Repo\robot-parameter-identification\identification\dynamics.py", line 422, in __get_regressor_objects
    self.reduced_regressor_for_id = Reduced_Regressor(os.path.join(cython_path, 'id'), full_regressor, state,
  File "C:\Users\Jonas\Repo\robot-parameter-identification\identification\dynamics.py", line 394, in __init__
    super().__init__(cython=True, cython_path=cython_path)
  File "C:\Users\Jonas\Repo\robot-parameter-identification\identification\dynamics.py", line 335, in __init__
    self._get_cy()
  File "C:\Users\Jonas\Repo\robot-parameter-identification\identification\dynamics.py", line 325, in _get_cy
    self.cy = autowrap(self.regressor, backend='cython', args=tuple(self._get_state()), verbose=True,
  File "C:\Users\Jonas\anaconda3\envs\robot-parameter-identification\lib\site-packages\sympy\core\cache.py", line 74, in wrapper
    retval = func(*args, **kwargs)
  File "C:\Users\Jonas\anaconda3\envs\robot-parameter-identification\lib\site-packages\sympy\utilities\autowrap.py", line 646, in autowrap
    return code_wrapper.wrap_code(routine, helpers=helps)
  File "C:\Users\Jonas\anaconda3\envs\robot-parameter-identification\lib\site-packages\sympy\utilities\autowrap.py", line 148, in wrap_code
    self._process_files(routine)
  File "C:\Users\Jonas\anaconda3\envs\robot-parameter-identification\lib\site-packages\sympy\utilities\autowrap.py", line 169, in _process_files
    raise CodeWrapError(
sympy.utilities.autowrap.CodeWrapError: Error while executing command: C:\Users\JonasStiegler\anaconda3\envs\robot-parameter-identification\python.exe setup.py build_ext --inplace. Command output is:
Compiling wrapper_module_0.pyx because it changed.
[1/1] Cythonizing wrapper_module_0.pyx
C:\Users\Jonas\anaconda3\envs\robot-parameter-identification\lib\site-packages\Cython\Compiler\Main.py:369: FutureWarning: Cython directive 'language_level' not set, using 2 for now (Py2). This will change in a later release! File: C:\Users\Jonas\Repo\robot-parameter-identification\AR1440_bugfix\regressor_files\id\wrapper_module_0.pyx
  tree = Parsing.p_module(s, pxd, full_module_name)
running build_ext
building 'wrapper_module_0' extension
creating build
creating build\temp.win-amd64-3.8
creating build\temp.win-amd64-3.8\Release
C:\Program Files (x86)\Microsoft Visual Studio\2019\BuildTools\VC\Tools\MSVC\14.28.29910\bin\HostX86\x64\cl.exe /c /nologo /Ox /W3 /GL /DNDEBUG /MD -IC:\Users\Jonas\anaconda3\envs\robot-parameter-identification\lib\site-packages\numpy\core\include -IC:\Users\Jonas\anaconda3\envs\robot-parameter-identification\include -IC:\Users\Jonas\anaconda3\envs\robot-parameter-identification\include "-IC:\Program Files (x86)\Microsoft Visual Studio\2019\BuildTools\VC\Tools\MSVC\14.28.29910\include" "-IC:\Program Files (x86)\Windows Kits\NETFXSDK\4.8\include\um" "-IC:\Program Files (x86)\Windows Kits\10\include\10.0.18362.0\ucrt" "-IC:\Program Files (x86)\Windows Kits\10\include\10.0.18362.0\shared" "-IC:\Program Files (x86)\Windows Kits\10\include\10.0.18362.0\um" "-IC:\Program Files (x86)\Windows Kits\10\include\10.0.18362.0\winrt" "-IC:\Program Files (x86)\Windows Kits\10\include\10.0.18362.0\cppwinrt" /Tcwrapper_module_0.c /Fobuild\temp.win-amd64-3.8\Release\wrapper_module_0.obj -std=c99
cl : Command line warning D9002 : ignoring unknown option '-std=c99'
wrapper_module_0.c
C:\Users\Jonas\anaconda3\envs\robot-parameter-identification\lib\site-packages\numpy\core\include\numpy\npy_1_7_deprecated_api.h(14) : Warning Msg: Using deprecated NumPy API, disable it with #define NPY_NO_DEPRECATED_API NPY_1_7_API_VERSION
C:\Program Files (x86)\Microsoft Visual Studio\2019\BuildTools\VC\Tools\MSVC\14.28.29910\bin\HostX86\x64\cl.exe /c /nologo /Ox /W3 /GL /DNDEBUG /MD -IC:\Users\Jonas\anaconda3\envs\robot-parameter-identification\lib\site-packages\numpy\core\include -IC:\Users\Jonas\anaconda3\envs\robot-parameter-identification\include -IC:\Users\Jonas\anaconda3\envs\robot-parameter-identification\include "-IC:\Program Files (x86)\Microsoft Visual Studio\2019\BuildTools\VC\Tools\MSVC\14.28.29910\include" "-IC:\Program Files (x86)\Windows Kits\NETFXSDK\4.8\include\um" "-IC:\Program Files (x86)\Windows Kits\10\include\10.0.18362.0\ucrt" "-IC:\Program Files (x86)\Windows Kits\10\include\10.0.18362.0\shared" "-IC:\Program Files (x86)\Windows Kits\10\include\10.0.18362.0\um" "-IC:\Program Files (x86)\Windows Kits\10\include\10.0.18362.0\winrt" "-IC:\Program Files (x86)\Windows Kits\10\include\10.0.18362.0\cppwinrt" /Tcwrapped_code_0.c /Fobuild\temp.win-amd64-3.8\Release\wrapped_code_0.obj -std=c99
cl : Command line warning D9002 : ignoring unknown option '-std=c99'
wrapped_code_0.c
creating C:\Users\Jonas\Repo\robot-parameter-identification\AR1440_bugfix\regressor_files\id\build\lib.win-amd64-3.8
C:\Program Files (x86)\Microsoft Visual Studio\2019\BuildTools\VC\Tools\MSVC\14.28.29910\bin\HostX86\x64\link.exe /nologo /INCREMENTAL:NO /LTCG /DLL /MANIFEST:EMBED,ID=2 /MANIFESTUAC:NO /LIBPATH:C:\Users\Jonas\anaconda3\envs\robot-parameter-identification\libs /LIBPATH:C:\Users\Jonas\anaconda3\envs\robot-parameter-identification\PCbuild\amd64 "/LIBPATH:C:\Program Files (x86)\Microsoft Visual Studio\2019\BuildTools\VC\Tools\MSVC\14.28.29910\lib\x64" "/LIBPATH:C:\Program Files (x86)\Windows Kits\NETFXSDK\4.8\lib\um\x64" "/LIBPATH:C:\Program Files (x86)\Windows Kits\10\lib\10.0.18362.0\ucrt\x64" "/LIBPATH:C:\Program Files (x86)\Windows Kits\10\lib\10.0.18362.0\um\x64" /EXPORT:PyInit_wrapper_module_0 build\temp.win-amd64-3.8\Release\wrapper_module_0.obj build\temp.win-amd64-3.8\Release\wrapped_code_0.obj /OUT:build\lib.win-amd64-3.8\wrapper_module_0.cp38-win_amd64.pyd /IMPLIB:build\temp.win-amd64-3.8\Release\wrapper_module_0.cp38-win_amd64.lib
   Creating library build\temp.win-amd64-3.8\Release\wrapper_module_0.cp38-win_amd64.lib and object build\temp.win-amd64-3.8\Release\wrapper_module_0.cp38-win_amd64.exp
Generating code
LINK : the 32-bit linker (C:\Program Files (x86)\Microsoft Visual Studio\2019\BuildTools\VC\Tools\MSVC\14.28.29910\bin\HostX86\x64\link.exe) ran out of heap space and is going to restart linking with a 64-bit linker
LINK : restarting link with 64-bit linker `C:\Program Files (x86)\Microsoft Visual Studio\2019\BuildTools\VC\Tools\MSVC\14.28.29910\bin\HostX64\x64\link.exe'
LINK : fatal error LNK1104: cannot open file 'build\lib.win-amd64-3.8\wrapper_module_0.cp38-win_amd64.pyd'
error: command 'C:\\Program Files (x86)\\Microsoft Visual Studio\\2019\\BuildTools\\VC\\Tools\\MSVC\\14.28.29910\\bin\\HostX86\\x64\\link.exe' failed with exit status 1104

Thanks in advance for all tips. If any needed information is missing, I will add if someone is asking for it.

Yunnosch
  • 26,130
  • 9
  • 42
  • 54
GroXXo
  • 23
  • 6
  • What did you change? Try reverting the input. If it works, that means the changes you've made aren't compatible with what you're trying to do. If it doesn't work, something else changed. Please include the sample data. – ewokx May 18 '21 at 08:20
  • It's quite likely that https://stackoverflow.com/a/65206777/4657412 is basically the same issue. I'm not completely sure how you'd apply it to sympy though – DavidW May 18 '21 at 08:31
  • @ewong It is a lot of code and overhead so it's going to be hard to include it here. However, using the older, smaller matrix does still work. My suspicion is that previously the 32-bit linker was enough but now it needs to restart with the 64-bit linker and this is where the problem occurs. – GroXXo May 18 '21 at 08:41
  • @DavidW thanks for the link, I will try it out! – GroXXo May 18 '21 at 08:42
  • To me it looks as if `build\lib.win-amd64-3.8\wrapper_module_0.cp38-win_amd64.pyd` is not accessible. Maybe it is already loaded or maybe still open by the 32bit-linker version. – ead May 18 '21 at 09:06
  • @ead yes, that was also my idea but I have no clue what I can do about it, except of forcing to use the 64bit linker from start on – GroXXo May 18 '21 at 09:59
  • @DavidW thanks for the help. I have found a solution and posted it as an answer. One thing I am unsure of is whether this is a bug in Sympy (which needs to be reported on their Github page) or if it was related to my machine only due to some weird setup. – GroXXo May 18 '21 at 10:46
  • Please do not edit solution announcements into the question. Accept (i.e. click the "tick" next to it) one of the existing answer, if there are any. You can also create your own answer, and even accept it, if your solution is not yet covered by an existing answer. – Yunnosch May 18 '21 at 10:56
  • @GroXXo It's ore likely a distutils problem than a sympy problem I think (I'm pretty sure I've seen this before). But I don't know how likely it is to be changed/fixed (understandably there's a fairly high barrier to changes to distutils) – DavidW May 18 '21 at 11:08
  • @GroXXo it is not a bug, but a conscious decision (see https://github.com/python/cpython/blob/bb3e0c240bc60fe08d332ff5955d54197f79751c/Lib/distutils/_msvccompiler.py#L160): "Always cross-compile from x86 to work with the lighter-weight MSVC installs that do not include native 64-bit tools." The issue is, that once 32bit linker failed, the file cannot be accessed by the 64bit linker -> it is an MSVC issue. – ead May 18 '21 at 11:38
  • @Yunnosch thanks for the hint! I'm new to StackOverflow so I wasn't sure what is the best way to mark it. I also tried accepting my own answer but apparently this can only be done 2 days after posting it – GroXXo May 18 '21 at 12:43
  • @ead thanks for the explanation! That makes a lot of sense – GroXXo May 18 '21 at 12:44

2 Answers2

1

With the help of @DavidW I was able to find a way around this issue, though it requires to modify Sympy's source code.

Background

Cython always tries to use the 32bit-linker first. When it runs out of heap, it will restart the conversion with the 64bit-linker. The problem here is that .pyd file that the 64bit-linker is not accessible most likely because it's either already loaded or maybe still opened by the 32bit-linker, as @ead pointed out.
By modifying the code as stated below, it is possible to force Cython to use the 64bit-linker from the beginning which avoids this problem.

Solution

  1. Open the autowrap.py file which is part of the Sympy package
  2. Go to class CythonCodeWrapper (line 218)
  3. Go to setup_template which is right below the class declaration
  4. Add the following two lines at the very start of the string:
from distutils import _msvccompiler
_msvccompiler.PLAT_TO_VCVARS['win-amd64'] = 'amd64'

The source code should now look like this:

class CythonCodeWrapper(CodeWrapper):
    """Wrapper that uses Cython"""

    setup_template = """\
from distutils import _msvccompiler
_msvccompiler.PLAT_TO_VCVARS['win-amd64'] = 'amd64'  
try:
    from setuptools import setup
    from setuptools import Extension
...
GroXXo
  • 23
  • 6
0

you could also drop the pyd file and recompile your code

slideWXY
  • 1,121
  • 8
  • 9