XGBoost installation issues for Python Anaconda Windows 10 (18 May 2018)

Question

Over the past several days I have tried to install XGBoost using instructions found at

Some of the instructions were straighforward (e.g., conda install -c conda-forge xgboost). Others involved a few dozen steps, some of which were unclear and confusing for a novice like me.

Some of the installations seemed to work, but importing the module in a jupyter notebook failed. For example, I can see installed files at ...\Anaconda2\envs\py36\Lib\site-packages\xgboost-0.71-py3.6.egg\xgboost, but importing produces an error.

My latest attempt followed instructions posted at https://www.kaggle.com/general/30163#latest-330213: conda install -c anaconda py-xgboost.

Again, the installation seemed to work: I can see the installed files under ...\Anaconda2\envs\py36\Lib\site-packages\xgboost. But in my notebook when I try to import the module using both

from xgboost import XGBRegressor

and

import xgboost

I get OSError: [WinError 126] The specified module could not be found error.

The full traceback is below.

Is there a fix for this? A better way to install? I'd like to continue with Dan Becker's intro to ML on kaggle!

VERSIONS:

 python version : 3.6.4150.1013
  conda version : 4.3.30

  Microsoft Windows : Version 10.0.16299.431
           platform : win-64
           git bash : launches a window labeled `MINGW64:/c/users/karls`

TRACEBACK:

---------------------------------------------------------------------------
OSError                                   Traceback (most recent call last)
<ipython-input-7-15d947748610> in <module>()
----> 1 from xgboost import XGBRegressor   # OSError: [WinError 126] The specified module could not be found
      2 
      3 my_model = XGBRegressor()
      4 # Add silent=True to avoid printing out updates with each cycle
      5 my_model.fit(train_X, train_y, verbose=False)

~\Anaconda2\envs\py36\lib\site-packages\xgboost\__init__.py in <module>()
      9 import os
     10 
---> 11 from .core import DMatrix, Booster
     12 from .training import train, cv
     13 from . import rabit                   # noqa

~\Anaconda2\envs\py36\lib\site-packages\xgboost\core.py in <module>()
    113 
    114 # load the XGBoost library globally
--> 115 _LIB = _load_lib()
    116 
    117 

~\Anaconda2\envs\py36\lib\site-packages\xgboost\core.py in _load_lib()
    107     if len(lib_path) == 0:
    108         return None
--> 109     lib = ctypes.cdll.LoadLibrary(lib_path[0])
    110     lib.XGBGetLastError.restype = ctypes.c_char_p
    111     return lib

~\Anaconda2\envs\py36\lib\ctypes\__init__.py in LoadLibrary(self, name)
    424 
    425     def LoadLibrary(self, name):
--> 426         return self._dlltype(name)
    427 
    428 cdll = LibraryLoader(CDLL)

~\Anaconda2\envs\py36\lib\ctypes\__init__.py in __init__(self, name, mode, handle, use_errno, use_last_error)
    346 
    347         if handle is None:
--> 348             self._handle = _dlopen(self._name, mode)
    349         else:
    350             self._handle = handle

OSError: [WinError 126] The specified module could not be found

EDIT 1 Install attempts

Example 1: conda install -c anaconda py-xgboost

Launch Anaconda prompt
Activate py36
Run conda install -c anaconda py-xgboost

Here are the command window results when I tried this install a few days ago (and below the "All requested packages already installed" confirmation when I retraced these steps today):

(py36) C:\Users\karls>conda install -c anaconda py-xgboost
Fetching package metadata ...............
Solving package specifications: .

Package plan for installation in environment C:\Users\karls\Anaconda2\envs\py36:

The following packages will be UPDATED:

    bzip2:        1.0.6-haa5b126_2        --> 1.0.6-vc14_3          anaconda [vc14]

The following packages will be SUPERSEDED by a higher-priority channel:

    cryptography: 2.1.4-py36he1d7878_0    --> 2.0.3-py36h123decb_1  anaconda
    curl:         7.58.0-h7602738_0       --> 7.55.1-vc14hdaba4a4_3 anaconda [vc14]
    freetype:     2.8-h51f8f2c_1          --> 2.8-vc14h17c9bdf_0    anaconda [vc14]
    hdf5:         1.10.1-h98b8871_1       --> 1.10.1-vc14hb361328_0 anaconda [vc14]
    icu:          58.2-ha66f8fd_1         --> 58.2-vc14hc45fdbb_0   anaconda [vc14]
    jpeg:         9b-hb83a4c4_2           --> 9b-vc14h4d7706e_1     anaconda [vc14]
    libiconv:     1.15-h1df5818_7         --> 1.15-vc14h29686d3_5   anaconda [vc14]
    libpng:       1.6.34-h79bbb47_0       --> 1.6.32-vc14h5163883_3 anaconda [vc14]
    libssh2:      1.8.0-hd619d38_4        --> 1.8.0-vc14hcf584a9_2  anaconda [vc14]
    libtiff:      4.0.9-h0f13578_0        --> 4.0.8-vc14h04e2a1e_10 anaconda [vc14]
    libxml2:      2.9.7-h79bbb47_0        --> 2.9.4-vc14h8fd0f11_5  anaconda [vc14]
    libxslt:      1.1.32-hf6f1972_0       --> 1.1.29-vc14hf85b8d4_5 anaconda [vc14]
    lxml:         4.2.1-py36heafd4d3_0    --> 4.1.1-py36he0adb16_0  anaconda
    lzo:          2.10-h6df0209_2         --> 2.10-vc14h0a64fa6_1   anaconda [vc14]
    openssl:      1.0.2n-h74b6da3_0       --> 1.0.2l-vc14hf4c37d5_5 anaconda [vc14]
    pillow:       5.0.0-py36h0738816_0    --> 4.2.1-py36hdb25ab2_0  anaconda
    py-xgboost:   0.71-py36h6538335_0     --> 0.71-py36h6538335_0   anaconda
    pycurl:       7.43.0.1-py36h74b6da3_0 --> 7.43.0-py36h086bf4c_3 anaconda
    pyopenssl:    17.5.0-py36h5b7d817_0   --> 17.4.0-py36h3316aad_0 anaconda
    qt:           5.6.2-vc14h6f8c307_12   --> 5.6.2-vc14h6f8c307_12 anaconda [vc14]
    sqlite:       3.22.0-h9d3ae62_0       --> 3.20.1-vc14h7ce8c62_1 anaconda [vc14]
    tk:           8.6.7-hcb92d03_3        --> 8.6.7-vc14hb68737d_1  anaconda [vc14]
    yaml:         0.1.7-hc54c509_2        --> 0.1.7-vc14hb31d195_1  anaconda [vc14]
    zlib:         1.2.11-h8395fce_2       --> 1.2.11-vc14h1cdd9ab_1 anaconda [vc14]

Proceed ([y]/n)?

I entered y, Enter and the module was installed. I didn't copy the results, but it indicated success -- but the moducle was not found when I tried to import it in a notebook.

Here are the results when I took these steps again today:

(C:\Users\karls\Anaconda2) C:\Users\karls>activate py36

(py36) C:\Users\karls>conda install -c anaconda py-xgboost
Fetching package metadata ...............
Solving package specifications: .

# All requested packages already installed.
# packages in environment at C:\Users\karls\Anaconda2\envs\py36:
#
py-xgboost                0.71             py36h6538335_0    anaconda

(py36) C:\Users\karls>

$Anaconda2\envs\py36\Lib\site-packages\xgboost$

$Anaconda2\envs\py36\Lib\site-packages\xgboost__pycache__$

EDIT 2 Example 2:

It looks like the actual C++ library can not be found in the installed python package. Could you explicitly the list of **all** steps that you have followed to reach the current state? — Mischa Lisovyi, May 20 '18 at 09:15
@Mykhailo Lisovyi, thanks for responding. I added my steps for one attempt at the bottom of my original post. — Karl Baker, May 22 '18 at 18:40
thanks. Since the problem seems to be due to the xgboost library not being found, maybe check that the `.dll` library is in `...\Anaconda2\envs\py36\Lib\site-packages\xgboost\lib` (or maybe on Windows the subdirectory is `Lib`). Also, these other thread might be helpful: https://stackoverflow.com/questions/35139108/how-to-install-xgboost-in-anaconda-python-windows-platform/35139185 https://stackoverflow.com/questions/36292484/xgboost-installation-issue-with-anaconda (note, that they uninstall `py-xgboost` before installation) — Mischa Lisovyi, May 23 '18 at 07:48
@Mykhailo Lisovyi, I have attached screenshots of the files in the xgboost dir at the bottom of my original post. I see no dll file. — Karl Baker, May 23 '18 at 21:17
@neigeldoi ok, that means that you have only the python binding installed, but on actuall underlying xgboost c++ library. so I would think that you would need to install `xgboost` together with `py-xgboost` package in conda. Were the two provided links of any help? — Mischa Lisovyi, May 23 '18 at 22:13
@Mykhailo Lisovyi, I looked at the various suggestions in the links you sent. It seemed like a lot of what I have seen and tried, though I haven't tried every single suggestion. This module seems different that all others I have installed, in that it seems to have a dozen or more potential install processes, only one of which may work on any given system. Is there a reason you know that xgboost does not install as seamlessly and painlessly as so many other packages? — Karl Baker, May 25 '18 at 04:14
@Mykhailo Lisovyi, regarding your suggestion to 'install `xgboost` together with `py-xgboost` package in conda,' I think I have. See Example 2 screenshots for `Anaconda2\envs\py36\Lib\site-packages\xgboost-0.71-py3.6.egg` at the end of my original post, the result of one of the other install attempts. — Karl Baker, May 25 '18 at 04:19
A disclaimer: I'm not an expert on anaconda or python package management. But from the past experience fighting very similar problems on Linux with a different package, the presence of the `.egg` directory does not help. Packages are searched by the name that you give in `import` in the `$PYTHONPATH`. Thus `.egg` doesn't help at all, as the folder name is different (after a quick google search, I think, those are packed version of a package for distribution). Once `pip install -U pack_name` helped me to install the egg into a full-blown package in `site-packages` — Mischa Lisovyi, May 25 '18 at 07:33
`xgboost` is actually not problematic to install. On Ubuntu itcomes in smooth. There are more problematic packages to deal with. In the worst (or most advanced, to be precise) case, you always have the opportunity to build the package by hand following instructions on the package web page. That's what also `conda install` or `pip install` are doing under the hood, but you are in full control of each step and see what fails. — Mischa Lisovyi, May 25 '18 at 07:36
@Mykhailo Lisovyi, your comment that `.egg` doesn't help gave me an idea. I did a simple copy/paste and the module is now importing. I copied the `xgboost` folder from inside the `.egg` folder and pasted it one level up, in the `site-packages` folder. (First I changed the name of the `xgboost` folder that was already in `site-packages` that wasn't working.) Ran my notebook and it seems to be working. First time I've dealt with eggs. Guess you could say we "cracked" it. Not sure it's supposed to work that way, but I'll take it. Many thanks for all your help. — Karl Baker, May 25 '18 at 09:08
NP. That's indeed not the intended way :) I imagine, this will cause problems for conda package update in your environment in the future. Maybe someone will suggest the proper procedure later on. Did you by chance try to do `pip install -U pack_name` before you did the copy? — Mischa Lisovyi, May 25 '18 at 13:25
@Mykhailo Lisovyi, `pip install -U xgboost` did not “unpack” the egg into `site-packages`. It says `Requirement already up-to-date: xgboost in ...\anaconda2\envs\py36\lib\site-packages\xgboost-0.71-py3.6.egg (0.71)`. It listed a few required upgrades (`numpy` and `scipy`) and installs (`terminado`, `ipykernel`), so I did those and tried `pip install -U xgboost` again. Same result, and it still lists numpy and scipy as “requirement not upgraded … in site-packages (from xgboost)” but does show the updated versions (1.14.3 and 1.1.0) that I installed. — Karl Baker, May 26 '18 at 03:12
The “(from xgboost)” clause in the “requirement not upgraded” sentence above makes me wonder: Should there be `numpy` and `scipy` files in the `… .egg\xgboost` dir? I notice it has several other `.py` files: `sklearn`, `plotting`, etc. I’d be interested to see a screenshot of a successful install showing all files in the `xgboost` dir and subdirs. — Karl Baker, May 26 '18 at 03:18
You can try using the installation instructions I included in [this post](https://stackoverflow.com/q/50557115/9374673). See if that solves your problems for you. — Mihai Chelaru, May 27 '18 at 22:46
Thanks, @MihaiChelaru. I'm not working in Visual Studio but will keep your process in mind if I do in future. — Karl Baker, May 28 '18 at 07:36
This is not an answer to your problem. But think critically if it is worth your time and effort. Why not load lightgbm (which is very similar to xgboost) and get on with your analysis?? — Regi Mathew, May 19 '19 at 01:30

Karl Baker · Accepted Answer · 2018-05-29T01:27:38.730

I found an install process that seems to be working in jupyter notebook with Anaconda 4.3 for python 3.6.4 on Windows 10 win-64. Below I spell out the process that I followed. At the bottom I include a couple screenshots of the installed folders and files. If you have any suggestions on how to improve this process, please let me know.

This process is adapted from instructions at http://adataanalyst.com/machine-learning/installing-xgboost-for-windows-10/ which in turn are derived from http://stackoverflow.com/questions/33749735/how-to-install-xgboost-package-in-python-windows-platform.

PREP

If you don’t have git, install it and add it to your PATH.
As part of previous attempts to install xgboost I had recently updated numpy and scipy to latest versions
Download and install MinGW-64: http://sourceforge.net/projects/mingw-w64/

a. In the Setting dialog, set the Architecture to “x86_64” (was i686) and the Threads to “win32” (was posix)

b. I installed MinGW-64 to the default file path in C:\Program Files, so I added C:\Program Files\mingw-w64\x86_64-8.1.0-win32-seh-rt_v6-rev0\mingw64\bin to my PATH environment variable

c. After installation finished, as suggested I went to the mingw64\bin folder and renamed mingw32-make to make
- Actually, I made a copy of mingw32-make and named the copy make
- Doing so may be the source of some of my troubles below, though I was able to get past them. I recommend you try renaming the file rather than leaving two copies of the same file with different names as I did

GET THE XGBOOST SOURCE CODE

Launch a Windows command prompt: Start | Windows System | Command Prompt
- These steps may also work in MINGW64, which I switch to later, but here I try to faithfully record the steps I took as I followed the instructions I had)
a. Enter cd c:\ where c:\ represents the location that you want to install xgboost. For me it was C:\...\Anaconda2\envs\py36\Lib\site-packages

b. Enter git clone --recursive https://github.com/dmlc/xgboost
- This will run and output a few dozen lines of output before displaying the prompt for the next step
c. Enter cd xgboost

d. Enter git submodule init
- This did not produce any output and immediately displayed the prompt again
e. Enter git submodule update
- This did not produce any output and immediately displayed the prompt again
f. Enter copy make\mingw64.mk config.mk
- Output: "1 file(s) copied."
- NOTE: Up to this point all commands were run in Windows command prompt. The next did not work there, so going forward I switched to the mingw64 terminal. I re-ran step "f" and continued at the next step.
Launch the mingw64 terminal: Start | MinGW--W64 project | Run terminal

a. Enter cd C:\Users\karls\Anaconda2\envs\py36\Lib\site-packages\xgboost

b. Enter copy make\mingw64.mk config.mk

c. Enter make -j4
- This command did not work. I tried dozens of variations based on suggested I googled: make.exe, makefile, cmake, pymake, make.py, mingw64-make, mingw64-make.exe, C:\Program Files\mingw-w64\x86_64-8.1.0-win32-seh-rt_v6-rev0\mingw64\bin\make, the list goes on. Nothing worked. I tried changing the directory to other folders inside of site-packages\xgboost. Finally, though the output was suspect, I hit upon the following and was able to proceed.
d. Change directory to ...\site-packages\xgboost\make

e. Enter mingw64.mk -j4
- This popped up a "How do you want to open this file?" dialog, which was the most hopeful output I had seen yet. I did not open the file. Did the command do any good? I have no idea, but I went on to the next steps.

INSTALL THE PYTHON PACKAGE

f. Change directory to site-packages\xgboost\python-package

g. Enter python setup.py install

This outputs several dozen lines: running this, creating that, writing and reading and copying, etc.
I also had several "warning: no files found matching ..." lines.
Presumably anything missing was dealt with in the followed lines of more writing, installing, running, creating, copying, byte-compiling, removing, processing, extracting, adding, and searching
The final line read "Finished processing dependencies for xgboost==0.71"

At this point I was able to import and use xgboost in a jupyter notebook, so I did not take any further steps. However, the instructions I was following (linked above) included additional steps that you may find necessary. Please let me know if you see any errors in my process as I am still suspect that my install is unsusceptible to problems with updates later (for example, the directory seems to contain copies of the same files in multiple places).

Below are a couple screenshots of the installed directories:

XGBoost installation issues for Python Anaconda Windows 10 (18 May 2018)

1 Answers1

Linked