How to write a solver.prototxt satisfy a given condition in CAFFE? - machine-learning

Question

I am writing the solver.prototxt that follows the rule of the paper https://arxiv.org/pdf/1604.02677.pdf
In the training phase, the learning rate was set as 0.001 initially and decreased by a factor of 10 when the loss stopped decreasing till 10−7. The discount weight was set as 1 initially and decreased by a factor of 10 every ten thousand iterations until a marginal value 10−3.
Note that, the discount weight is loss_weight in Caffe. Based on the information above, I wrote my solver as
train_net: "train.prototxt"
lr_policy: "step"
gamma: 0.1
stepsize: 10000
base_lr: 0.001 #0.002
In train.prototxt, I also set
layer {
name: "loss"
type: "SoftmaxWithLoss"
bottom: "deconv"
bottom: "label"
top: "loss"
loss_weight: 1
}
However, I still don't know how to set solver to satisfy the rule "decreased by a factor of 10 when the loss stopped decreasing till 10−7" and "decreased by a factor of 10 every ten thousand iterations until a marginal value 10−3". I did not found any caffe rule can do it as the reference:
// The learning rate decay policy. The currently implemented learning rate
// policies are as follows:
// - fixed: always return base_lr.
// - step: return base_lr * gamma ^ (floor(iter / step))
// - exp: return base_lr * gamma ^ iter
// - inv: return base_lr * (1 + gamma * iter) ^ (- power)
// - multistep: similar to step but it allows non uniform steps defined by
// stepvalue
// - poly: the effective learning rate follows a polynomial decay, to be
// zero by the max_iter. return base_lr (1 - iter/max_iter) ^ (power)
// - sigmoid: the effective learning rate follows a sigmod decay
// return base_lr ( 1/(1 + exp(-gamma * (iter - stepsize))))
//
// where base_lr, max_iter, gamma, step, stepvalue and power are defined
// in the solver parameter protocol buffer, and iter is the current iteration.
If anyone knows it, please give me some guide to writing the solver.prototxt to satisfy above condition.

Answer 1

Learning rate reduction
Part of the problem is that the phrase decreased by a factor of 10 when the loss stopped decreasing till 10e−7 does not quite make sense. I think that, perhaps, the authors are trying to say that every time the loss quit decreasing, they reduced the learning rate by a factor of 10, until the learning rate got to 10e-7.
If so, then this is a manual process, not something you can choose with Caffe parameters. Most of all, "when the loss stopped decreasing" is a non-trivial judgement, although a long-base moving average will give you a good indication. I expect that the authors did this manually, stopping and restarting the training from a checkpoint.
You can get a similar effect with a learning rate decay policy of step: set gamma to 0.1, and set the step parameter high enough to ensure that training has levelled off before each rate reduction. This will waste some computer time, but might save you overall trouble.
Discount weight
In Caffe, the loss weight is merely the relative weighting among the various losses in the model, linear factors used to achieve the final loss statistic. Caffe does not provide run-time alteration of the weight. Perhaps this was something else that the authors tuned by hand.
I tried reading the areas of the paper around the two references to "discount weight", but found it hard reading. I'll wait until someone proofreads and edits that paper for grammar and clarity. In the meantime, I hope this answer helps you some.
You can find a little more information here.