XGB via Scikit learn API doesn't seem to be running in GPU although compiled to run for GPU - machine-learning

Question

It appears although XGB is compiled to run on GPU, when called/executed via Scikit learn API, it doesn't seem to be running on GPU.
Please advise if this is expected behaviour

Answer 1

As far as I can tell, the Scikit learn API does not currently support GPU. You need to use the learning API (e.g. xgboost.train(...)). This also requires you to first convert your data into xgboost DMatrix.
Example:
params = {"updater":"grow_gpu"}
train = xgboost.DMatrix(x_train, label=y_train)
clf = xgboost.train(params, train, num_boost_round=10)
UPDATE:
The Scikit Learn API now supports GPU via the **kwargs argument:
http://xgboost.readthedocs.io/en/latest/python/python_api.html#id1

Answer 2

I couldn't get this working from the pip installed XGBoost, but I pulled the most recent XGBoost from GitHub (git clone --recursive https://github.com/dmlc/xgboost) and compiled it with the PLUGIN_UPDATER_GPU flag which allowed me to use the GPU with the sklearn API. This required me to also change some NVCC flags to work on my GTX960 that was causing some build errors, then some runtime errors due to architecture mismatch. After it built, I installed with pip install -e python-package/ within the repo directory. To use the Scikit learn API (using either grow_gpu or grow_hist_gpu):
import xgboost as xgb
model = xgb.XGBClassifier(
max_depth=5,
objective='binary:logistic',
**{"updater": "grow_gpu"}
)
model.fit(train_x, train_y)
If anyone is interested in the process to fix the build with the GPU flag, here is the process that I went through on Ubuntu 14.04.
i) git clone git clone --recursive https://github.com/dmlc/xgboost
ii) cd insto xgboost and make -j4 to create multi-threaded, if no GPU is desired
iii) to make GPU, edit make/config.mk to use PLUGIN_UPDATER_GPU
iv) Edit the makefile Makefile, on the NVCC section to use the flag --gpu-architecture=sm_xx for GPU version (5.2 for GTX 960) on line 101
#CODE = $(foreach ver,$(COMPUTE),-gencode arch=compute_$(ver),code=sm_$(ver)) TO
CODE = --gpu-architecture=sm_52
v) Run the ./build.sh, it should say completed in multi-threaded mode or the NVCC build probably failed (or another error, look above for the error)
vi) In the virtualenv (if desired) in the same directory run pip install -e python-package/
These are some things that caused some nvcc errors for me:
i) Installing/updating the Cuda Toolkit by downloading the cuda toolkit .deb from Nvidia (version 8.0 worked for me, and is required in some cases?).
ii) Install/update cuda
sudo apt-get update
sudo apt-get install cuda
iii) Add nvcc to your path. Mine was in /usr/local/cuda/bin/
iv) A restart may be required if running nvidia-smi does not work due to some of the cuda/driver/toolkit updates.