I followed the instructions provided in this link: How to install xgboost package in python (windows platform)? and https://xgboost.readthedocs.io/en/latest/build.html#installing-r-package-with-gpu-support
My setup: Windows 10 Home x64. Installed Visual 2015, Rtools (make sure the PATH has following two items. PATH: c:\Rtools\bin;c:\Rtools\mingw_64\bin), Cmake and Git Bash.
I setup R to run in CMD and executed git bash as administrator.
Step 1 Clone xgboost source
$ git clone https://github.com/dmlc/xgboost
the above package is installed in the path: *C:\Users\Username*
Step 2 Install xgboost
cd xgboost
git submodule init
git submodule update
cd R-package
R CMD INSTALL .
Then the package installed successfully in C:\Program Files\R\R-3.5.1\library.
Step 3 Installing R package with GPU support
mkdir build
cd build
cmake .. -G"Visual Studio 14 2015 Win64" -DUSE_CUDA=ON -DR_LIB=ON -DLIBR_EXECUTABLE="C:/Program Files/R/R-3.4.1/bin/x64/R.exe"
Then I installed the build using the following command
cmake --build . --target install --config Release
After this, the libraries are updated in C:\Program Files\R\R-3.5.1\library with GPU support. Run RStudio as administrator to access the library.
Source: Fail to install R XGBoost with GPU support on Windows 7