I am using optuna to tune xgboost model's hyperparameters. I find it stuck at trial 2 (trial_id=3) for a long time(244 minutes). But When I look at the SQLite database which records the trial data, I find all the trial 2 (trial_id=3) hyperparameters has been calculated except the mean squared error value of trial 2. And the optuna trial 2 (trial_id=3) seems stuck at that step. I want to know why this happened? And how to fix the issue?
Here is the code
def xgb_hyperparameter_tuning():
def objective(trial):
params = {
"n_estimators": trial.suggest_int("n_estimators", 1000, 10000, step=100),
"booster": trial.suggest_categorical("booster", ["gbtree", "gblinear", "dart"]),
"max_depth": trial.suggest_int("max_depth", 1, 20, step=1),
"learning_rate": trial.suggest_float("learning_rate", 0.0001, 0.2, step=0.001),
"min_child_weight": trial.suggest_float("min_child_weight", 1.0, 20.0, step=1.0),
"colsample_bytree": trial.suggest_float("colsample_bytree", 0.1, 1.0, step=0.1),
"subsample": trial.suggest_float("subsample",0.1, 1.0, step=0.1),
"reg_alpha": trial.suggest_float("reg_alpha", 0.0, 11.0, step=0.1),
"reg_lambda": trial.suggest_float("reg_lambda", 0.0, 11.0, step=0.1),
"num_parallel_tree": 10,
"random_state": 16,
"n_jobs": 10,
"early_stopping_rounds": 1000,
}
model = XGBRegressor(**params)
mse = make_scorer(mean_squared_error)
cv = cross_val_score(estimator=model, X=X_train, y=log_y_train, cv=20, scoring=mse, n_jobs=-1)
return cv.mean()
study = optuna.create_study(study_name="HousePriceCompetitionXGB", direction="minimize", storage="sqlite:///house_price_competition_xgb.db", load_if_exists=True)
study.optimize(objective, n_trials=100,)
return None
xgb_hyperparameter_tuning()
Here is the output
[I 2021-11-16 10:06:27,522] A new study created in RDB with name: HousePriceCompetitionXGB
[I 2021-11-16 10:08:40,050] Trial 0 finished with value: 0.03599314763859092 and parameters: {'n_estimators': 5800, 'booster': 'gblinear', 'max_depth': 4, 'learning_rate': 0.1641, 'min_child_weight': 17.0, 'colsample_bytree': 0.4, 'subsample': 0.30000000000000004, 'reg_alpha': 10.8, 'reg_lambda': 7.6000000000000005}. Best is trial 0 with value: 0.03599314763859092.
[I 2021-11-16 10:11:55,830] Trial 1 finished with value: 0.028514652199592445 and parameters: {'n_estimators': 6600, 'booster': 'gblinear', 'max_depth': 17, 'learning_rate': 0.0821, 'min_child_weight': 20.0, 'colsample_bytree': 0.7000000000000001, 'subsample': 0.2, 'reg_alpha': 1.2000000000000002, 'reg_lambda': 7.2}. Best is trial 1 with value: 0.028514652199592445.
Here is the sqlite database trial_values table's data
trial_value_id
trial_id
objective
value
1
1
0
0.0359931476385909
2
2
0
0.0285146521995924
Here is the sqlite database trial_params table's data And you can see all the trial 2 (trial_id=3) hyperparameters has been calculated
param_id
trial_id
param_name
param_value
distribution_json
1
1
n_estimators
5800.0
{"name": "IntUniformDistribution", "attributes": {"low": 1000, "high": 10000, "step": 100}}
2
1
booster
1.0
{"name": "CategoricalDistribution", "attributes": {"choices": ["gbtree", "gblinear", "dart"]}}
3
1
max_depth
4.0
{"name": "IntUniformDistribution", "attributes": {"low": 1, "high": 20, "step": 1}}
4
1
learning_rate
0.1641
{"name": "DiscreteUniformDistribution", "attributes": {"low": 0.0001, "high": 0.1991, "q": 0.001}}
5
1
min_child_weight
17.0
{"name": "DiscreteUniformDistribution", "attributes": {"low": 1.0, "high": 20.0, "q": 1.0}}
6
1
colsample_bytree
0.4
{"name": "DiscreteUniformDistribution", "attributes": {"low": 0.1, "high": 1.0, "q": 0.1}}
7
1
subsample
0.3
{"name": "DiscreteUniformDistribution", "attributes": {"low": 0.1, "high": 1.0, "q": 0.1}}
8
1
reg_alpha
10.8
{"name": "DiscreteUniformDistribution", "attributes": {"low": 0.0, "high": 11.0, "q": 0.1}}
9
1
reg_lambda
7.6
{"name": "DiscreteUniformDistribution", "attributes": {"low": 0.0, "high": 11.0, "q": 0.1}}
10
2
n_estimators
6600.0
{"name": "IntUniformDistribution", "attributes": {"low": 1000, "high": 10000, "step": 100}}
11
2
booster
1.0
{"name": "CategoricalDistribution", "attributes": {"choices": ["gbtree", "gblinear", "dart"]}}
12
2
max_depth
17.0
{"name": "IntUniformDistribution", "attributes": {"low": 1, "high": 20, "step": 1}}
13
2
learning_rate
0.0821
{"name": "DiscreteUniformDistribution", "attributes": {"low": 0.0001, "high": 0.1991, "q": 0.001}}
14
2
min_child_weight
20.0
{"name": "DiscreteUniformDistribution", "attributes": {"low": 1.0, "high": 20.0, "q": 1.0}}
15
2
colsample_bytree
0.7
{"name": "DiscreteUniformDistribution", "attributes": {"low": 0.1, "high": 1.0, "q": 0.1}}
16
2
subsample
0.2
{"name": "DiscreteUniformDistribution", "attributes": {"low": 0.1, "high": 1.0, "q": 0.1}}
17
2
reg_alpha
1.2
{"name": "DiscreteUniformDistribution", "attributes": {"low": 0.0, "high": 11.0, "q": 0.1}}
18
2
reg_lambda
7.2
{"name": "DiscreteUniformDistribution", "attributes": {"low": 0.0, "high": 11.0, "q": 0.1}}
19
3
n_estimators
7700.0
{"name": "IntUniformDistribution", "attributes": {"low": 1000, "high": 10000, "step": 100}}
20
3
booster
2.0
{"name": "CategoricalDistribution", "attributes": {"choices": ["gbtree", "gblinear", "dart"]}}
21
3
max_depth
4.0
{"name": "IntUniformDistribution", "attributes": {"low": 1, "high": 20, "step": 1}}
22
3
learning_rate
0.1221
{"name": "DiscreteUniformDistribution", "attributes": {"low": 0.0001, "high": 0.1991, "q": 0.001}}
23
3
min_child_weight
3.0
{"name": "DiscreteUniformDistribution", "attributes": {"low": 1.0, "high": 20.0, "q": 1.0}}
24
3
colsample_bytree
0.5
{"name": "DiscreteUniformDistribution", "attributes": {"low": 0.1, "high": 1.0, "q": 0.1}}
25
3
subsample
0.1
{"name": "DiscreteUniformDistribution", "attributes": {"low": 0.1, "high": 1.0, "q": 0.1}}
26
3
reg_alpha
10.8
{"name": "DiscreteUniformDistribution", "attributes": {"low": 0.0, "high": 11.0, "q": 0.1}}
27
3
reg_lambda
1.1
{"name": "DiscreteUniformDistribution", "attributes": {"low": 0.0, "high": 11.0, "q": 0.1}}
Although I am not 100% sure, I think I know what happened.
This issue happens because some parameters are not suitable for certain booster type and the trial will return nan as result and be stuck at the step - calculating the MSE score.
To solve the problem, you just need to delete the "booster": "dart".
In other words, using "booster": trial.suggest_categorical("booster", ["gbtree", "gblinear"]), rather than "booster": trial.suggest_categorical("booster", ["gbtree", "gblinear", "dart"]), can solve the problem.
I got the idea when I tuned my LightGBMRegressor Model. I found many trials fail because these trials returned nan and they all used the same "boosting_type"="rf". So I deleted the rf and all 100 trials were completed without any error. Then I looked for the XGBRegressor issue which I posted above. I found all the trials which were stuck had the same "booster":"dart" either. So I deleted the dart, and the XGBRegressor run normally.
Related
I was a bit puzzled to find the GeoJSON
{
"type": "Polygon",
"coordinates": [
[
[-0.8, -0.8],
[ 0.8, -0.8],
[ 0.8, 0.8],
[-0.8, 0.8],
[-0.8, -0.8]
],
[
[-0.6, -0.6],
[-0.6, 0.6],
[ 0.6, 0.6],
[ 0.6, -0.6],
[-0.6, -0.6]
],
[
[-0.4, -0.4],
[ 0.4, -0.4],
[ 0.4, 0.4],
[-0.4, 0.4],
[-0.4, -0.4]
]
]
}
an "island in a lake in an island" if you will, rejected by one online validator, but accepted by another.
Looking again at RFC7846
For Polygons with more than one of these rings, the first MUST be
the exterior ring, and any others MUST be interior rings. The
exterior ring bounds the surface, and the interior rings (if
present) bound holes within the surface.
it looks to me that the first validator was correct, the final ring does not bound a hole in the surface defined by the first ring, so is invalid. So am I right in thinking that a valid GeoJSON Polygon cannot represent an island in a lake in an island? (So one would need to use a MultiPolygon to represent it?)
Correct, the inner rings are holes in the exterior ring so cannot have an island in a lake where an inner ring is not a hole.
A valid representation of an island would be a MultiPolygon where the first polygon is the larger polygon with a hole that forms the lake and the second polygon is the smaller polygon inside the first which is the island.
Here is the GeoJSON:
{
"type": "Feature",
"geometry": {
"type": "MultiPolygon",
"coordinates": [
[
[
[-0.8, -0.8],
[ 0.8, -0.8],
[ 0.8, 0.8],
[-0.8, 0.8],
[-0.8, -0.8]
],
[
[-0.6, -0.6],
[-0.6, 0.6],
[ 0.6, 0.6],
[ 0.6, -0.6],
[-0.6, -0.6]
]
], [
[
[-0.4, -0.4],
[ 0.4, -0.4],
[ 0.4, 0.4],
[-0.4, 0.4],
[-0.4, -0.4]
]
]
]
}
}
I have a project I'm working on that involves rendering 3D models in WebGL, GitHub here. In pulling together several different resources, I've found two different formats for the model data: one with JSON entries like so:
var houseIFS =
{
"vertices": [
[ 2, -1, 2 ],
[ 2, -1, -2 ],
[ 2, 1, -2 ],
[ 2, 1, 2 ],
[ 1.5, 1.5, 0 ],
[ -1.5, 1.5, 0 ],
[ -2, -1, 2 ],
[ -2, 1, 2 ],
[ -2, 1, -2 ],
[ -2, -1, -2 ]
],
"faces": [
[ 0, 1, 2, 3 ],
[ 3, 2, 4 ],
[ 7, 3, 4, 5 ],
[ 2, 8, 5, 4 ],
[ 5, 8, 7 ],
[ 0, 3, 7, 6 ],
[ 0, 6, 9, 1 ],
[ 2, 1, 9, 8 ],
[ 6, 7, 8, 9 ]
],
"normals": [
[ 1, 0, 0 ],
[ 0.7071, 0.7071, 0 ],
[ 0, 0.9701, 0.2425 ],
[ 0, 0.9701, -0.2425 ],
[ -0.7071, 0.7071, 0 ],
[ 0, 0, 1 ],
[ 0, -1, 0 ],
[ 0, 0, -1 ],
[ -1, 0, 0 ]
],
"faceColors": [
[ 1, .8, .8 ],
[ .7, .7, 1 ],
[ 0, 0, 1 ],
[ 0, 0, .7 ],
[ .7, .7, 1 ],
[ 1, 0, 0 ],
[ .4, .4, .4 ],
[ 1, 0, 0 ],
[ 1, .8, .8 ],
]
};
and another with more primitive return types:
/** The return value of each function is an object, model, with properties:
*
* model.vertexPositions -- the vertex coordinates;
* model.vertexNormals -- the normal vectors;
* model.vertexTextureCoords -- the texture coordinates;
* model.indices -- the face indices.
*
* The first three properties are of type Float32Array, while
* model.indices is of type Uint16Array.
*/
I tried to create a method to convert the data from the "modern" version to the "primitive":
function convertPoly(model) {
return {
vertexPositions: new Float32Array(model.vertices),
vertexNormals: new Float32Array(model.normals),
vertexTextureCoords: new Float32Array(model.faces),
indices: new Uint16Array(model.faces)
}
}
but I don't think this is correct, and I don't see anything rendered after trying to render it. How can I compute indices from the vertices or faces? I guess I don't really understand what the indices really represent or how they work (is it triangle vertices of the faces?).
I am currently working with Highcharts in combination with the pattern fill module. When I set a pattern for a series in the chart, the pattern is shown but it has a transparent background. I need to set an additional background because the pattern is overlapping with another series which I don't want to see behind it. You can check this fiddle. So basically I don't want to see those three columns on the left behind the pattern. Any ideas how I can do that? I haven't seen any options to set an additional background, but maybe you know some trick. This is the code I am using for the pattern:
"color": {
"pattern": {
"path": {
"d": "M 0 0 L 10 10 M 9 -1 L 11 1 M -1 9 L 1 11"
},
"width": 10,
"height": 10,
"opacity": 1,
"color": "rgb(84,198,232)"
}
}
You need to set fill attribute as a path property:
"color": {
"pattern": {
"path": {
"d": "M 0 0 L 10 10 M 9 -1 L 11 1 M -1 9 L 1 11",
fill: 'red'
},
"width": 10,
"height": 10,
"opacity": 1,
"color": 'rgb(84,198,232)'
}
}
Live demo: https://jsfiddle.net/BlackLabel/m9rxwej5/
I guess there's been an update. backgroundColor should be set at pattern's root level:
"color": {
"pattern": {
"backgroundColor": 'red',
"path": {
"d": "M 0 0 L 10 10 M 9 -1 L 11 1 M -1 9 L 1 11",
},
"width": 10,
"height": 10,
"opacity": 1,
"color": 'rgb(84,198,232)',
}
}
https://jsfiddle.net/vL4fqhao/
I try to understand the data in the BoxTextured Model for the TEXCOORD_0 accessor.
As seen in the capture, the datas seems correct for POSITION and NORMALS but why values in the TEXCOORD_0 accessor aren't in range of "max": [ 1.0, 1.0 ], "min": [ 0.0, 0.0 ] but have a "max": [ 6.0, 1.0 ] ?
{
"bufferView": 2,
"byteOffset": 0,
"componentType": 5126,
"count": 24,
"max": [
6.0,
1.0
],
"min": [
0.0,
0.0
],
"type": "VEC2"
}
Should those be normalized ?
My texture applied is totally wrong : Rendered with uv test texture.
Where is my misunderstanding ?
Thank you
(I know I have a problem with my face orientation but that's another problem)
The 6.0 comes from the number of faces on the cube. Note that the sampler specifies REPEAT (10497):
"samplers": [
{
"magFilter": 9729,
"minFilter": 9986,
"wrapS": 10497,
"wrapT": 10497
}
],
so the image will be tiled repeatedly. It's just a simple way to get the logo rendered on all six faces of the cube.
I want to name a few points on my time data with irregular intervals. I am trying this as shown on fiddle by appending as
{ [Date.UTC(1970, 9, 27), 0 ] , name: 'point 1'},
but it is not working, any inputs ? I also want to have color for those points.
DEMO
You will have to pass data like:
data: [
{x:Date.UTC(1970, 11, 2), y:2,color:'red', name:'point 1'},
{x:Date.UTC(1970, 11, 3), y:3,color:'blue', name:'point 2'},
{x:Date.UTC(1970, 11, 4), y:4,color:'orange', name:'point 3'},
[Date.UTC(1970, 11, 2), 0.8 ],
[Date.UTC(1970, 11, 9), 0.6 ],
.......
...
Change from:
{ [Date.UTC(1970, 9, 27), 0 ] , name: 'point 1'},
To:
{ x: Date.UTC(1970, 9, 27), y: 0, name: 'point 1'},