Training caffe library and Loss does not converge - machine-learning

Question

I use caffe for my recognition and I have an issue that loss data never converge.
My training parameters in the configuration are
Conf.base_lr = 0.2;
Conf.max_iter = 800001;
Conf.test_iter = 100;
Conf.test_interval = 1000;
Conf.momentum = 0.5;
Conf.random_seed = 2;
Conf.clip_gradients = 0.1;
Conf.gamma = 0.8;
Conf.stepsize = 100000;
Conf.weights = "";
//logging
Conf.display_interval = 100;
Conf.snapshot_prefix_folder = "../tmp";
Conf.snapshot_interval = 10000;
Conf.schematic_path = "../tmp/reinspect.png";
Conf.graph_prefix = "../tmp/history";
Conf.log_file = "../tmp/log_brainwash.txt";
Conf.graph_interval = 500;
//net
Conf.init_range = 0.1;
Then when I check Backward data
All net params (data, diff): L1 norm = (208684, 3.43485e+11); L2 norm = (135.231, 3.96399e+08)
Diff values of L1 and L2 norms are huge and not normal. What could be wrong with my parameters in configuration and how to tune them?
Some of my log data for forward and backward can be seen in this link.
EDIT:
Previously I have problem of some layers are not included in the Backward propagation. So now I force them and all are included except these with no bottom layers like Input and DummyData. They are shown below.
This development is similar implementation to this Lib (only Python and C++ are different). They include all those layers in Backward propagation, DummyData is NumpyData in Python in their implementation. If necessary, how to include those into Backward propagation.
layer {
name: "image"
type: "Input"
top: "image"
input_param { shape: { dim: 1 dim: 3 dim: 480 dim: 640 } }
}
layer {
name: "box_flags"
type: "Input"
top: "box_flags"
input_param { shape: { dim: 300 dim: 1 dim: 5 dim: 1 } }
}
layer {
name: "boxes"
type: "Input"
top: "boxes"
input_param { shape: { dim: 300 dim: 4 dim: 5 dim: 1 } }
}
layer {
name: "lstm_hidden_seed"
type: "DummyData"
top: "lstm_hidden_seed"
dummy_data_param {
shape { dim: 300 dim: 250 }
data_filler { type: "constant" value: 0 }
}
}
layer {
name: "lstm_mem_seed"
type: "DummyData"
top: "lstm_mem_seed"
dummy_data_param {
shape { dim: 300 dim: 250 }
data_filler { type: "constant" value: 0 }
}
}
DummyData layer was NumpyData when it was in Python, when I convert to C++, it is changed to DummyData with initialization data 0.
Do I need to include all those Input and DummyData into Backward propagation?
I still have this abnormal big values at L1 and L2 norm.
[Backward] All net params (data, diff): L1 norm = (208696, 4.09333e+06); L2 norm = (135.23, 4791.7)

Answer 1

Your learning rate is very high, making your optimization process diverge. Try reduce it by factor of at least 50 and re-start the training.