Is log loss in XGBoost and Sklearn the same? - machine-learning

Question

I'm playing around with a new data set with XGBoost. Following is my code:
import xgboost as xgb
import pandas as pd
import numpy as np
train = pd.read_csv("train_users_processed_onehot.csv")
labels = train["Buy"].map({"Y":1, "N":0})
features = train.drop("Buy", axis=1)
data_dmat = xgb.DMatrix(data=features, label=labels)
params={"max_depth":5, "min_child_weight":2, "eta": 0.1, "subsamples":0.9, "colsample_bytree":0.8, "objective" : "binary:logistic", "eval_metric": "logloss", "seed": 2333}
rounds = 6000
result = xgb.cv(params=params, dtrain=data_dmat, num_boost_round=rounds, early_stopping_rounds=50, as_pandas=True, seed=2333)
print result
The result is (omitted intermediate results):
test-logloss-mean test-logloss-std train-logloss-mean
0 0.683354 0.000058 0.683206
165 0.622318 0.000661 0.607680
But when I'm trying to do parameter tuning with GridSearchCV, I found the result to be quite different. To be more specific, this is my code:
import xgboost as xgb
from sklearn.model_selection import GridSearchCV
from xgboost.sklearn import XGBClassifier
import numpy as np
import pandas as pd
train_dataframe = pd.read_csv("train_users_processed_onehot.csv")
train_labels = train_dataframe["Buy"].map({"Y":1, "N":0})
train_features = train_dataframe.drop("Buy", axis=1)
params = {"max_depth": [5], "min_child_weight": [2]}
estimator = XGBClassifier(learning_rate=0.1, n_estimators=170, max_depth=2, min_child_weight=4, objective="binary:logistic", subsample=0.9, colsample_bytree=0.8, seed=2333)
gsearch1 = GridSearchCV(estimator, param_grid=params, n_jobs=4, iid=False, verbose=1, scoring="neg_log_loss")
gsearch1.fit(train_features.values, train_labels.values)
print pd.DataFrame(gsearch1.cv_results_)
print gsearch1.best_params_
print -gsearch1.best_score_
and I got:
mean_fit_time mean_score_time mean_test_score mean_train_score
0 87.71497 0.209772 -3.134132 -0.567306
It is clear that 3.134132 is very different from 0.622318. What's the reason of this?
Thank you!

Answer 1

You pass different parameters to the both:
max_depth: 5 vs 2
eta: 0.1 vs 0.3 (default)
min_child_weight: 2 vs 4
The parameters you pass to sklearn are more conservative (it's less likely you will overfit the model), so the algorithm does not try too much to fit the model to the data. In turn, on second you get a lower score - exactly what should be expected.