Changing order of imports results in error in Python

Question

I'm trying to understand the cause of the following error. First if I type the following into python

>>> import scipy.sparse
>>> import torch

it runs without error. However, when I type in in

>>> import torch
>>> import scipy.sparse

I get the following error:
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/global/software/sl-7.x86_64/modules/langs/python/3.6/lib/python3.6/site-packages/scipy/sparse/__init__.py", line 229, in <module>
    from .csr import *
  File "/global/software/sl-7.x86_64/modules/langs/python/3.6/lib/python3.6/site-packages/scipy/sparse/csr.py", line 15, in <module>
    from ._sparsetools import csr_tocsc, csr_tobsr, csr_count_blocks, \
ImportError: /lib64/libstdc++.so.6: version `CXXABI_1.3.9' not found (required by /global/software/sl-7.x86_64/modules/langs/python/3.6/lib/python3.6/site-packages/scipy/sparse/_sparsetools.cpython-36m-x86_64-linux-gnu.so)

I can even go the directory "/global/software/sl-7.x86_64/modules/langs/python/3.6/lib/python3.6/site-packages/scipy/sparse/" and import the binary "_sparsetools.cpython-36m-x86_64-linux-gnu.so" followed by torch without issue. But if I try it the other way around I again get the above error.

Does anyone have any idea why changing the order of these imports should have a different effect?

Answer 1

The simple search strategy for shared objects assumes that a single version of each exists—or at least that directories containing newer versions are put first on the search path. The path includes $LD_LIBRARY_PATH (which should be avoided), DT_RPATH and its newer variant DT_RUNPATH (which crucially depend on the client being loaded), and system directories like /lib. This works well for systems following the FHS with global package management, but packages that are installed, with copies of their dependencies, in a single per-package directory (as is common on Windows and with some “normal user” package managers) can easily produce multiple versions of a shared object with the same soname.

The expectation is that sharing that name is harmless because one can be used in place of the other (and therefore be put first on the path). The reality is that there is no single directory that is newest for all libraries, and there’s not even a single path to configure given the DT_ tags.

The result is that whichever one is loaded first wins: the dynamic loader can’t load both of them (since they provide many of the same symbols), so the second request has only the effect of checking the library version tags, which here are found to be inadequate. In this case, one client (torch) is relying on the system’s C++ standard library, while the other (_sparsetools) has its own, newer version. It may or may not even need its newer version: since it was built against it, it is conservatively marked as needing it by default.

This is a hard problem: not even tools like virtual environments or environment modules can handle it, since the problem lies in the incompatible compilation of extant packages. Systems that rebuild everything from source (like Nix or Spack) can, but only at their usual cost of controlling all relevant builds. It may be that simply controlling the import order is, unfortunately, the best choice available.

Answer 2

@DavisHerring's answer gives you an explanation what is going on, here is a possible workaround to ensure that the right version is loaded - the LD_PRELOAD-trick:

1. Step:

Find the right libc++.so-version via console:

$ ldd _sparsetools.cpython-36m-x86_64-linux-gnu.so
libstdc++.so.6 => <path to right version>/libstdc++.so.6(...)

2. Step:

While starting python, preload the right version, so the loader picks the right version:

$ LD_PRELOAD=<path to right version>/libstdc++.so python

---

However the best solution would to rebuild pytorch with the right libc++-dependency.