You're running into a quite common problem in the Cygwin world. There are many URLs mentioning (dealing with) it, but I'm going to list the ones that I came across:
- [SO]: Cygwin error: “-bash: fork: retry: Resource temporarily unavailable”
- [SO]: Cygwin issue - unable to remap; same address as parent
- [SuperUser]: Cygwin fatal error unable to remap.. What does it mean?
- [WordPress]: Cygwin and Rails – unable to remap to same address as parent; died waiting for dll loading, errno 11
- [SO]: Cygwin error: “child_info_fork::abort: Loaded to different address:”
The "behind the scenes magic" is very well explained in [Cygwin]: Problems with process creation (emphases are mine):
The semantics of fork require that a forked child process have exactly the same address space layout as its parent. However, Windows provides no native support for cloning address space between processes and several features actively undermine a reliable fork implementation. Three issues are especially prevalent:
- DLL base address collisions. Unlike *nix shared libraries, which use "position-independent code", Windows shared libraries assume a fixed base address. Whenever the hard-wired address ranges of two DLLs collide (which occurs quite often), the Windows loader must "rebase" one of them to a different address. However, it may not resolve collisions consistently, and may rebase a different dll and/or move it to a different address every time. Cygwin can usually compensate for this effect when it involves libraries opened dynamically, but collisions among statically-linked dlls (dependencies known at compile time) are resolved before
cygwin1.dll initializes and cannot be fixed afterward. This problem can only be solved by removing the base address conflicts which cause the problem, usually using the rebaseall tool.
- Address space layout randomization (ASLR). Starting with Vista, Windows implements ASLR, which means that thread stacks, heap, memory-mapped files, and statically-linked dlls are placed at different (random) locations in each process. This behaviour interferes with a proper
fork, and if an unmovable object (process heap or system dll) ends up at the wrong location, Cygwin can do nothing to compensate (though it will retry a few times automatically).
A troubleshooter is attempted in [Cygwin.FAQ]: 4.45. How do I fix fork() failures? (emphases still mine). With the risk of spamming the answer, I'm going to paste it here:
Unfortunately, Windows does not use the fork/exec model of process creation found in UNIX-like OSes, so it is difficult for Cygwin to implement a reliable and correct fork(), which can lead to error messages such as:
- unable to remap somedll to same address as parent
- couldn't allocate heap
- died waiting for dll loading
- child -1 - died waiting for longjmp before initialization
- STATUS_ACCESS_VIOLATION
- resource temporarily unavailable
Potential solutions for the above errors:
Restart whatever process is trying (and failing) to use fork(). Sometimes Windows sets up a process environment that is even more hostile to fork() than usual.
Ensure that you have eliminated (not just disabled) all software on the BLODA.
Switch from 32-bit Cygwin to 64-bit Cygwin, if your OS and CPU support that. With the bigger address space fork() is less likely to fail.
Try setting the environment variable CYGWIN to "detect_bloda", which enables some extra debugging, which may indicate what other software is causing the problem.
See this mail for more information.
Force a full rebase: Run rebase-trigger fullrebase, exit all Cygwin programs and run Cygwin setup.
By default, Cygwin's setup program automatically performs an incremental rebase of newly installed files. Forcing a full rebase causes the rebase map to be cleared before doing the rebase.
See /usr/share/doc/rebase/README and /usr/share/doc/Cygwin/_autorebase.README for more details.
Please note that installing new packages or updating existing ones undoes the effects of rebase and often causes fork() failures to reappear.
See the process creation section of the User's Guide for the technical reasons it is so difficult to make fork() work reliably.
In order to reproduce the problem, I used:
- Cygwin 32:
- The chance of running into the issue is much higher
- It's not my main Cygwin env
- Python 3.6.4 + VEnv
- Located at /home/cfati/Work/Dev/VEnvs/py_032_03.06.04_test0
I tried reproducing you exact behavior (with _lbfgsb*.dll), but pip -v install scipy failed to build it.
Since [GitHub.SciPy]: Installing SciPy on Windows describes a quite complex process, and I'd have no guarantee that at the end I'd be able to reproduce the problem, I tried with numpy's .dlls (numpy was successfully installed as a scipy dependency), but I wasn't able to (as a side effect, import numpy loads a bunch of .dlls), but calling fork (via subprocess.Popen) didn't fail.
Then I decided to take matter into my own hands, and create a small program that loads some .dlls, then forks itself (again, via subprocess.Popen), to make the problem as reproducible as possible.
dll.c:
#include <stdio.h>
#if defined(_WIN32)
# define DLL_EXPORT __declspec(dllexport)
#else
# define DLL_EXPORT
#endif
DLL_EXPORT int test() {
printf("[%s] (%d) - [%s]\n", __FILE__, __LINE__, __FUNCTION__);
}
code.py:
#!/usr/bin/env python3
import sys
import os
import subprocess
import time
import select
import random
import ctypes
DLLS = [os.path.join(os.path.dirname(__file__), "dll{:d}.dll".format(item)) for item in range(2)]
def main():
random.seed(os.getpid())
random.shuffle(DLLS)
if len(sys.argv) == 1:
print("Python {:s} on {:s}\n".format(sys.version.replace("\n", ""), sys.platform))
print("Process 0x{:08X}".format(os.getpid()))
for dll in DLLS:
ctypes.cdll.LoadLibrary(dll)
idx = 0
while sys.stdin not in select.select([sys.stdin], [], [], 1)[0]:
p = subprocess.Popen([sys.executable] + sys.argv + [str(idx)])
#p.communicate()
time.sleep(1)
idx += 1
else:
sleep_time = 3
print("Process 0x{:08X} (inner) will end in {:d} seconds".format(os.getpid(), sleep_time))
time.sleep(sleep_time)
if __name__ == "__main__":
main()
Notes:
- The scenario is a bit different, instead of a Python Extension Module, I have a regular (dummy) .dll, which I attempt loading it via [Python.Docs]: ctypes - A foreign function library for Python. This should make no difference, as no matter how Python sees it (as an module, or as an external .dll) it will still have to load it into the process (the same way)
- The scenario is:
- I load 2 such .dlls (actually it's the same .dll copied under a different name, so they both have the same Preferred Base Address), so the 1st .dll will probably be loaded at that address, while the next .dll will be loaded at a different one
- Then I fork the process, and in the child, based on a random factor, I switch the .dll loading order
- When the 2nd .dll will be loaded at the preferred base, there'll be an inconsistency with the parent process, yielding the error
- At the beginning everything was in my cwd, then (to be closer to your problem), I created a Python package with the files. Note that I didn't do it the proper way (installation via a setup.py file), but manually copying everything
[cfati@cfati-5510-0:/cygdrive/e/Work/Dev/StackOverflow/q054370263]> ~/sopr.sh
*** Set shorter prompt to better fit when pasted in StackOverflow (or other) pages ***
[prompt]>
[prompt]> uname -a
CYGWIN_NT-10.0-WOW cfati-5510-0 2.11.2(0.329/5/3) 2018-11-08 14:30 i686 Cygwin
[prompt]> ls
code.py dll.c scipy.txt
[prompt]> # List the currently installed packages in the !!! VEENV !!! Python
[prompt]> ls -l ~/Work/Dev/VEnvs/py_032_03.06.04_test0/lib/python3.6/site-packages
total 33
drwxr-xr-x+ 1 cfati None 0 Jan 30 01:40 __pycache__
-rw-r--r-- 1 cfati None 126 Jan 30 01:40 easy_install.py
drwxr-xr-x+ 1 cfati None 0 Feb 2 21:41 numpy
drwxr-xr-x+ 1 cfati None 0 Feb 2 21:41 numpy-1.16.1.dist-info
drwxr-xr-x+ 1 cfati None 0 Jan 30 01:40 pip
drwxr-xr-x+ 1 cfati None 0 Jan 30 01:40 pip-19.0.1.dist-info
drwxr-xr-x+ 1 cfati None 0 Jan 30 01:40 pkg_resources
drwxr-xr-x+ 1 cfati None 0 Jan 30 01:40 setuptools
drwxr-xr-x+ 1 cfati None 0 Jan 30 01:40 setuptools-40.7.1.dist-info
drwxr-xr-x+ 1 cfati None 0 Jan 30 01:40 wheel
drwxr-xr-x+ 1 cfati None 0 Jan 30 01:40 wheel-0.32.3.dist-info
[prompt]>
[prompt]> ~/Work/Dev/VEnvs/py_032_03.06.04_test0/bin/python -m q054370263
/home/cfati/Work/Dev/VEnvs/py_032_03.06.04_test0/bin/python: No module named q054370263
[prompt]> # Create the package in site-packages dir
[prompt]> mkdir ~/Work/Dev/VEnvs/py_032_03.06.04_test0/lib/python3.6/site-packages/q054370263
[prompt]> cp code.py ~/Work/Dev/VEnvs/py_032_03.06.04_test0/lib/python3.6/site-packages/q054370263/__main__.py
[prompt]> gcc -fPIC -shared -o ~/Work/Dev/VEnvs/py_032_03.06.04_test0/lib/python3.6/site-packages/q054370263/dll0.dll dll.c
[prompt]> cp ~/Work/Dev/VEnvs/py_032_03.06.04_test0/lib/python3.6/site-packages/q054370263/dll0.dll ~/Work/Dev/VEnvs/py_032_03.06.04_test0/lib/python3.6/site-packages/q054370263/dll1.dll
[prompt]> ls
code.py dll.c scipy.txt
[prompt]> ls -l ~/Work/Dev/VEnvs/py_032_03.06.04_test0/lib/python3.6/site-packages/q054370263
total 260
-rwxr-x--- 1 cfati None 1012 Feb 3 12:39 __main__.py
-rwxr-xr-x 1 cfati None 129844 Feb 3 12:22 dll0.dll
-rwxr-xr-x 1 cfati None 129844 Feb 3 12:22 dll1.dll
[prompt]>
[prompt]> # Attempt to reproduce the problem by simply running the package
[prompt]> ~/Work/Dev/VEnvs/py_032_03.06.04_test0/bin/python -m q054370263
Python 3.6.4 (default, Jan 7 2018, 17:45:56) [GCC 6.4.0] on cygwin
Process 0x00001B38
18 [main] python3 21616 child_info_fork::abort: address space needed by 'dll0.dll' (0xD90000) is already occupied
Traceback (most recent call last):
File "/usr/lib/python3.6/runpy.py", line 193, in _run_module_as_main
"__main__", mod_spec)
File "/usr/lib/python3.6/runpy.py", line 85, in _run_code
exec(code, run_globals)
File "/home/cfati/Work/Dev/VEnvs/py_032_03.06.04_test0/lib/python3.6/site-packages/q054370263/__main__.py", line 37, in <module>
main()
File "/home/cfati/Work/Dev/VEnvs/py_032_03.06.04_test0/lib/python3.6/site-packages/q054370263/__main__.py", line 25, in main
p = subprocess.Popen([sys.executable] + sys.argv + [str(idx)])
File "/usr/lib/python3.6/subprocess.py", line 709, in __init__
restore_signals, start_new_session)
File "/usr/lib/python3.6/subprocess.py", line 1275, in _execute_child
restore_signals, start_new_session, preexec_fn)
BlockingIOError: [Errno 11] Resource temporarily unavailable
[prompt]>
[prompt]> ~/Work/Dev/VEnvs/py_032_03.06.04_test0/bin/python -m q054370263
Python 3.6.4 (default, Jan 7 2018, 17:45:56) [GCC 6.4.0] on cygwin
Process 0x000055E8
Process 0x00005764 (inner) will end in 3 seconds
1 [main] python3 21224 child_info_fork::abort: address space needed by 'dll1.dll' (0x6D0000) is already occupied
Traceback (most recent call last):
File "/usr/lib/python3.6/runpy.py", line 193, in _run_module_as_main
"__main__", mod_spec)
File "/usr/lib/python3.6/runpy.py", line 85, in _run_code
exec(code, run_globals)
File "/home/cfati/Work/Dev/VEnvs/py_032_03.06.04_test0/lib/python3.6/site-packages/q054370263/__main__.py", line 37, in <module>
main()
File "/home/cfati/Work/Dev/VEnvs/py_032_03.06.04_test0/lib/python3.6/site-packages/q054370263/__main__.py", line 25, in main
p = subprocess.Popen([sys.executable] + sys.argv + [str(idx)])
File "/usr/lib/python3.6/subprocess.py", line 709, in __init__
restore_signals, start_new_session)
File "/usr/lib/python3.6/subprocess.py", line 1275, in _execute_child
restore_signals, start_new_session, preexec_fn)
BlockingIOError: [Errno 11] Resource temporarily unavailable
So, the error is quite reproducible. I'd also like to add here the .dll details (Dependency Walker):
Why doesn't a regular rebase (rebaseall) solve the problem?
Cygwin packages have a post install script that calls rebase on their .dlls
rebase searches in standard library paths (/lib, /usr/lib, ...). That can be adjusted, according to /usr/share/doc/Cygwin/_autorebase.README:
Packages can make the potential locations of such dynamic objects
known by dropping a file (named after the package) in
/var/lib/rebase/dynpath.d/. If any dynamic objects are installed by
users, these locations should be advertised in /var/lib/rebase/user.d/
(the file name should be identical to the user name if there are
multiple users on this system)
Python requires such an adjustment, due to packages that may contain .dlls
pip like packages do not have a post install script (that would rebase their .dlls)
The VEnv is in the user's home path which is not in the standard library paths (so, even a rebaseall will ignore them)
Note that all the rebased .dlls are stored in a DB: /etc/rebase.db(.${ARCH}).
[prompt]> ls /var/lib/rebase/dynpath.d/
perl python2 python3
[prompt]> cat /var/lib/rebase/dynpath.d/python3
/usr/lib/python3.6/site-packages
[prompt]> ls /var/lib/rebase/user.d/
[prompt]>
[prompt]> grep -r "dll0.dll" /etc/rebase.db.i386
[prompt]>
In order the .dlls to be picked up by rebase tools, they need to be advertised. That can be done in 2 ways:
Specify them in one of the custom locations, so at the next full rebase they will no longer be ignored (simply add the VEnv dir, and others if any):
[prompt]> echo /home/cfati/Work/Dev/VEnvs/py_032_03.06.04_test0 >/var/lib/rebase/user.d/${USER}
[prompt]> cat /var/lib/rebase/user.d/cfati
/home/cfati/Work/Dev/VEnvs/py_032_03.06.04_test0
Manually rebase the .dlls
Both of them worked for me, but I'm going to exemplify on the 2nd variant only (as it's simpler). The process consists of 2 steps:
The updated Dependency Walker window (check its Preferred Base, and compare it to the one in the previous image):
And also the rebase DB "query" (now back from Mintty):
[prompt]> grep -r "dll0.dll" /etc/rebase.db.i386
Binary file /etc/rebase.db.i386 matches
What's more important, running the code:
[prompt]> ~/Work/Dev/VEnvs/py_032_03.06.04_test0/bin/python -m q054370263
Python 3.6.4 (default, Jan 7 2018, 17:45:56) [GCC 6.4.0] on cygwin
Process 0x000052D0
Process 0x00004634 (inner) will end in 3 seconds
Process 0x00004864 (inner) will end in 3 seconds
Process 0x00005CFC (inner) will end in 3 seconds
Process 0x00005A5C (inner) will end in 3 seconds
Process 0x00005098 (inner) will end in 3 seconds
Process 0x00005840 (inner) will end in 3 seconds
Process 0x000058C4 (inner) will end in 3 seconds
Process 0x000051DC (inner) will end in 3 seconds
Process 0x00001A5C (inner) will end in 3 seconds
Process 0x00003D2C (inner) will end in 3 seconds
Process 0x00000DA0 (inner) will end in 3 seconds
As seen it is able to fork itself multiple times (it only stopped because I pressed Enter).
