How to prevent weight update in caffe - machine-learning

Question

Some layers of my net load a pretrained model. I want to fix their parameters and train other layers.
I followed this page and set lr_multi decay_multi to 0, propagate_down: false, and even base_lr: 0 weight_decay: 0 in the solver. However, the test loss (use all test images for each test) is still changing very slowly in each iter. After thousands iters the accuracy would go to 0 (from 80% when the pre-trained model was loaded).
Here is a a two-layer example which I just initialize the weights and set above parameters to 0. I want to feeze all layers in this example but when training start, the loss keeps changing...
layer {
name: "data"
type: "ImageData"
top: "data"
top: "label"
include {
phase: TRAIN
}
transform_param {
scale: 0.017
mirror: true
crop_size: 32
mean_value: 115
mean_value: 126
mean_value: 130
color: true
contrast: true
brightness: true
}
image_data_param {
source: "/data/zhuhao5/data/cifar100/cifar100_train_replicate.txt"
batch_size: 64
shuffle: true
#pair_size: 3
}
}
layer {
name: "data"
type: "ImageData"
top: "data"
top: "label"
include {
phase: TEST
}
transform_param {
scale: 0.017
mirror: false
crop_size: 32
mean_value: 115
mean_value: 126
mean_value: 130
}
image_data_param {
source: "/data/zhuhao5/data/cifar100/cifar100_test.txt"
batch_size: 100
shuffle: false
}
}
#-------------- TEACHER --------------------
layer {
name: "conv1"
type: "Convolution"
bottom: "data"
propagate_down: false
top: "conv1"
param {
lr_mult: 0
decay_mult: 0
}
convolution_param {
num_output: 16
bias_term: false
pad: 1
kernel_size: 3
stride: 1
weight_filler {
type: "msra"
}
}
}
layer {
name: "res2_1a_1_bn"
type: "BatchNorm"
bottom: "conv1"
propagate_down: false
top: "res2_1a_1_bn"
param {
lr_mult: 0
decay_mult: 0
}
param {
lr_mult: 0
decay_mult: 0
}
}
layer {
name: "res2_1a_1_scale"
type: "Scale"
bottom: "res2_1a_1_bn"
propagate_down: false
top: "res2_1a_1_bn"
param {
lr_mult: 0
decay_mult: 0
}
scale_param {
bias_term: true
}
}
layer {
name: "res2_1a_1_relu"
type: "ReLU"
bottom: "res2_1a_1_bn"
propagate_down: false
top: "res2_1a_1_bn"
}
layer {
name: "pool_5"
type: "Pooling"
bottom: "res2_1a_1_bn"
propagate_down: false
top: "pool_5"
pooling_param {
pool: AVE
global_pooling: true
}
}
layer {
name: "fc100"
type: "InnerProduct"
bottom: "pool_5"
propagate_down: false
top: "fc100"
param {
lr_mult: 0
decay_mult: 0
}
param {
lr_mult: 0
decay_mult: 0
}
inner_product_param {
num_output: 100
weight_filler {
type: "msra"
}
bias_filler {
type: "constant"
value: 0
}
}
}
#---------------------------------
layer {
name: "tea_soft_loss"
type: "SoftmaxWithLoss"
bottom: "fc100"
bottom: "label"
propagate_down: false
propagate_down: false
top: "tea_soft_loss"
loss_weight: 0
}
##----------- ACCURACY----------------
layer {
name: "teacher_accuracy"
type: "Accuracy"
bottom: "fc100"
bottom: "label"
top: "teacher_accuracy"
accuracy_param {
top_k: 1
}
}
Here is the solver:
test_iter: 100
test_interval: 10
base_lr: 0
momentum: 0
weight_decay: 0
lr_policy: "poly"
power: 1
display: 10000
max_iter: 80000
snapshot: 5000
type: "SGD"
solver_mode: GPU
random_seed: 10086
and log:
I0829 16:31:39.363433 14986 net.cpp:200] teacher_accuracy does not need backward computation.
I0829 16:31:39.363438 14986 net.cpp:200] tea_soft_loss does not need backward computation.
I0829 16:31:39.363442 14986 net.cpp:200] fc100_fc100_0_split does not need backward computation.
I0829 16:31:39.363446 14986 net.cpp:200] fc100 does not need backward computation.
I0829 16:31:39.363451 14986 net.cpp:200] pool_5 does not need backward computation.
I0829 16:31:39.363454 14986 net.cpp:200] res2_1a_1_relu does not need backward computation.
I0829 16:31:39.363458 14986 net.cpp:200] res2_1a_1_scale does not need backward computation.
I0829 16:31:39.363462 14986 net.cpp:200] res2_1a_1_bn does not need backward computation.
I0829 16:31:39.363466 14986 net.cpp:200] conv1 does not need backward computation.
I0829 16:31:39.363471 14986 net.cpp:200] label_data_1_split does not need backward computation.
I0829 16:31:39.363485 14986 net.cpp:200] data does not need backward computation.
I0829 16:31:39.363490 14986 net.cpp:242] This network produces output tea_soft_loss
I0829 16:31:39.363494 14986 net.cpp:242] This network produces output teacher_accuracy
I0829 16:31:39.363507 14986 net.cpp:255] Network initialization done.
I0829 16:31:39.363559 14986 solver.cpp:56] Solver scaffolding done.
I0829 16:31:39.363852 14986 caffe.cpp:248] Starting Optimization
I0829 16:31:39.363862 14986 solver.cpp:272] Solving WRN_22_12_to_WRN_18_4_v5_net
I0829 16:31:39.363865 14986 solver.cpp:273] Learning Rate Policy: poly
I0829 16:31:39.365981 14986 solver.cpp:330] Iteration 0, Testing net (#0)
I0829 16:31:39.366190 14986 blocking_queue.cpp:49] Waiting for data
I0829 16:31:39.742347 14986 solver.cpp:397] Test net output #0: tea_soft_loss = 85.9064
I0829 16:31:39.742437 14986 solver.cpp:397] Test net output #1: teacher_accuracy = 0.0113
I0829 16:31:39.749806 14986 solver.cpp:218] Iteration 0 (0 iter/s, 0.385886s/10000 iters), loss = 0
I0829 16:31:39.749862 14986 solver.cpp:237] Train net output #0: tea_soft_loss = 4.97483
I0829 16:31:39.749877 14986 solver.cpp:237] Train net output #1: teacher_accuracy = 0
I0829 16:31:39.749908 14986 sgd_solver.cpp:105] Iteration 0, lr = 0
I0829 16:31:39.794306 14986 solver.cpp:330] Iteration 10, Testing net (#0)
I0829 16:31:40.171447 14986 solver.cpp:397] Test net output #0: tea_soft_loss = 4.9119
I0829 16:31:40.171510 14986 solver.cpp:397] Test net output #1: teacher_accuracy = 0.0115
I0829 16:31:40.219133 14986 solver.cpp:330] Iteration 20, Testing net (#0)
I0829 16:31:40.596911 14986 solver.cpp:397] Test net output #0: tea_soft_loss = 4.91862
I0829 16:31:40.596971 14986 solver.cpp:397] Test net output #1: teacher_accuracy = 0.0116
I0829 16:31:40.645246 14986 solver.cpp:330] Iteration 30, Testing net (#0)
I0829 16:31:41.021711 14986 solver.cpp:397] Test net output #0: tea_soft_loss = 4.92105
I0829 16:31:41.021772 14986 solver.cpp:397] Test net output #1: teacher_accuracy = 0.0117
I0829 16:31:41.069464 14986 solver.cpp:330] Iteration 40, Testing net (#0)
I0829 16:31:41.447345 14986 solver.cpp:397] Test net output #0: tea_soft_loss = 4.91916
I0829 16:31:41.447407 14986 solver.cpp:397] Test net output #1: teacher_accuracy = 0.0117
I0829 16:31:41.495157 14986 solver.cpp:330] Iteration 50, Testing net (#0)
I0829 16:31:41.905607 14986 solver.cpp:397] Test net output #0: tea_soft_loss = 4.9208
I0829 16:31:41.905654 14986 solver.cpp:397] Test net output #1: teacher_accuracy = 0.0117
I0829 16:31:41.952659 14986 solver.cpp:330] Iteration 60, Testing net (#0)
I0829 16:31:42.327942 14986 solver.cpp:397] Test net output #0: tea_soft_loss = 4.91936
I0829 16:31:42.328025 14986 solver.cpp:397] Test net output #1: teacher_accuracy = 0.0117
I0829 16:31:42.374279 14986 solver.cpp:330] Iteration 70, Testing net (#0)
I0829 16:31:42.761359 14986 solver.cpp:397] Test net output #0: tea_soft_loss = 4.91859
I0829 16:31:42.761430 14986 solver.cpp:397] Test net output #1: teacher_accuracy = 0.0117
I0829 16:31:42.807821 14986 solver.cpp:330] Iteration 80, Testing net (#0)
I0829 16:31:43.232321 14986 solver.cpp:397] Test net output #0: tea_soft_loss = 4.91668
I0829 16:31:43.232398 14986 solver.cpp:397] Test net output #1: teacher_accuracy = 0.0117
I0829 16:31:43.266436 14986 solver.cpp:330] Iteration 90, Testing net (#0)
I0829 16:31:43.514633 14986 blocking_queue.cpp:49] Waiting for data
I0829 16:31:43.638617 14986 solver.cpp:397] Test net output #0: tea_soft_loss = 4.91836
I0829 16:31:43.638684 14986 solver.cpp:397] Test net output #1: teacher_accuracy = 0.0117
I0829 16:31:43.685451 14986 solver.cpp:330] Iteration 100, Testing net (#0)
I wonder what I missed in the updating process in caffe :(

Answer 1

Found the reason.
BatchNorm layer use different use_global_stats in TRAIN and TEST phases.
In my issue, I should set use_global_stats: true in the training process.
And also don't forget the Scale layer.
The revised layers should be
layer {
name: "res2_1a_1_bn"
type: "BatchNorm"
bottom: "conv1"
top: "res2_1a_1_bn"
batch_norm_param {
use_global_stats: true
}
}
layer {
name: "res2_1a_1_scale"
type: "Scale"
bottom: "res2_1a_1_bn"
top: "res2_1a_1_bn"
param {
lr_mult: 0
decay_mult: 0
}
param {
lr_mult: 0
decay_mult: 0
}
scale_param {
bias_term: true
}
}