The benefit using in_place=True for a layer is for saving memory. However, I found some paper like ResNet that use in_place=False for the BatchNorm layer. My question is when do we use in_place=True/False for the BatchNorm layer. Does it has any effect on performance? Thanks all
layer {
name: "bnorm1"
type: "BatchNorm"
bottom: "conv1_bn"
top: "conv1_bn"
batch_norm_param {
use_global_stats: false
}
}
The related code to make above prototxt is
n.bnorm1 = L.BatchNorm(n.conv1_bn, batch_norm_param=dict(use_global_stats=False), in_place=True)
n.scale1 = L.Scale(n.bnorm1, in_place=True, bias_term=True,filler=dict(value=1), bias_filler=dict(value=0))
Related
I managed to create a CoreML 2.0 model with flexible input/output shape sizes:
I can't figure out how to set the size in my Xcode project, however. If I set the input pixel buffer size 2048x2048, the output pixel buffer is still 1536x1536. If I set it to 768x768, the resulting pixel buffer is still 1536x1536 - but is blank outside the region of 768x768.
I examined the automatically generated Swift model class and don't see any clues there.
I can't find a single example anywhere showing how to use the "Flexibility" sizes.
In the WWDC 2018 Session 708 "What's New in Core ML", Part 1 it states:
This means that now you have to ship a single model. You don't have to have any redundant code. And if you need to switch between standard definition and high definition, you can do it much faster because we don't need to reload the model from scratch; we just need to resize it. You have two options to specify the flexibility of the model. You can define a range for its dimension, so you can define a minimal width and height and the maximum width and height. And then at inference pick any value in between. But there is also another way. You can enumerate all the shapes that you are going to use. For example, all different aspect ratios, all different resolutions, and this is better for performance. Core ML knows more about your use case earlier, so it can -- it has the opportunities of performing more optimizations.
They say "we just need to resize it". It so frustrating because they don't tell you how to just resize it! They also say "And then at inference pick any value in between" but offer no clue how to pick the value in between!
Here is how I added the flexible shape sizes:
import coremltools
from coremltools.models.neural_network import flexible_shape_utils
spec = coremltools.utils.load_spec('mymodel_fxedShape.mlmodel')
img_size_ranges = flexible_shape_utils.NeuralNetworkImageSizeRange()
img_size_ranges.add_height_range(640, 2048)
img_size_ranges.add_width_range(640, 2048)
flexible_shape_utils.update_image_size_range(spec, feature_name='inputImage', size_range=img_size_ranges)
flexible_shape_utils.update_image_size_range(spec, feature_name='outputImage', size_range=img_size_ranges)
coremltools.utils.save_spec(spec, 'myModel.mlmodel')
Here is the description of the model:
description {
input {
name: "inputImage"
shortDescription: "Image to stylize"
type {
imageType {
width: 1536
height: 1536
colorSpace: BGR
imageSizeRange {
widthRange {
lowerBound: 640
upperBound: 2048
}
heightRange {
lowerBound: 640
upperBound: 2048
}
}
}
}
}
output {
name: "outputImage"
shortDescription: "Stylized image"
type {
imageType {
width: 1536
height: 1536
colorSpace: BGR
imageSizeRange {
widthRange {
lowerBound: 640
upperBound: 2048
}
heightRange {
lowerBound: 640
upperBound: 2048
}
}
}
}
}
}
There are two layers using "outputShape":
layers {
name: "SpatialFullConvolution_63"
input: "Sequential_53"
output: "SpatialFullConvolution_63_output"
convolution {
outputChannels: 16
kernelChannels: 32
nGroups: 1
kernelSize: 3
kernelSize: 3
stride: 2
stride: 2
dilationFactor: 1
dilationFactor: 1
valid {
paddingAmounts {
borderAmounts {
}
borderAmounts {
}
}
}
isDeconvolution: true
hasBias: true
weights {
}
bias {
}
outputShape: 770
outputShape: 770
}
}
...relu layer...
layers {
name: "SpatialFullConvolution_67"
input: "ReLU_66"
output: "SpatialFullConvolution_67_output"
convolution {
outputChannels: 8
kernelChannels: 16
nGroups: 1
kernelSize: 3
kernelSize: 3
stride: 2
stride: 2
dilationFactor: 1
dilationFactor: 1
valid {
paddingAmounts {
borderAmounts {
}
borderAmounts {
}
}
}
isDeconvolution: true
hasBias: true
weights {
}
bias {
}
outputShape: 1538
outputShape: 1538
}
}
I am now trying to figure out how to remove the outputShape from those two layers.
>>> layer = spec.neuralNetwork.layers[49]
>>> layer.convolution.outputShape
[1538L, 1538L]
I tried setting it to []:
layer.convolution.outputShape = []
To a Shape:
layer.convolution.outputShape = flexible_shape_utils.Shape(())
Whatever I try, I get the error:
TypeError: Can't set composite field
Do I have to create a new layer and then link it to the layer that is outputting to it and the layer it is outputting to?
The issue in this case was that there were layers present in the model that used a fixed shape for their outputShapes. For example:
>>> layer = spec.neuralNetwork.layers[49]
>>> layer.convolution.outputShape
[1538L, 1538L]
The model in question was indeed fully convolutional, so before converting to CoreML, it worked with any input and output shapes.
I was able to delete the fixed outputShape with this command:
layer = spec.neuralNetwork.layers[49]
del layer.convolution.outputShape[:]
After doing that, the model worked with flexible input and output shapes.
All credit for this answer goes to Matthijs Hollemans.
I trained a model with 4 hidden layers and 2 dense layers and I have saved that model.
Now I want to load that model and want to split into two models, one with one hidden layers and another one with only dense layers.
I have splitted the model with hidden layer in the following way
model = load_model ("model.hdf5")
HL_model = Model(inputs=model.input, outputs=model.layers[7].output)
Here the model is loaded model, in the that 7th layer is my last hidden layer. I tried to split the dense in the like
DL_model = Model(inputs=model.layers[8].input, outputs=model.layers[-1].output)
and I am getting error
TypeError: Input layers to a `Model` must be `InputLayer` objects.
After splitting, the output of the HL_model will the input for the DL_model.
Can anyone help me to create a model with dense layer?
PS :
I have tried below code too
from keras.layers import Input
inputs = Input(shape=(9, 9, 32), tensor=model_1.layers[8].input)
model_3 = Model(inputs=inputs, outputs=model_1.layers[-1].output)
And getting error as
RuntimeError: Graph disconnected: cannot obtain value for tensor Tensor("conv2d_1_input:0", shape=(?, 144, 144, 3), dtype=float32) at layer "conv2d_1_input". The following previous layers were accessed without issue: []
here (144, 144, 3) in the input image size of the model.
You need to specify a new Input layer first, then stack the remaining layers over it:
DL_input = Input(model.layers[8].input_shape[1:])
DL_model = DL_input
for layer in model.layers[8:]:
DL_model = layer(DL_model)
DL_model = Model(inputs=DL_input, outputs=DL_model)
A little more generic. You can use the following function to split a model
from keras.layers import Input
from keras.models import Model
def get_bottom_top_model(model, layer_name):
layer = model.get_layer(layer_name)
bottom_input = Input(model.input_shape[1:])
bottom_output = bottom_input
top_input = Input(layer.output_shape[1:])
top_output = top_input
bottom = True
for layer in model.layers:
if bottom:
bottom_output = layer(bottom_output)
else:
top_output = layer(top_output)
if layer.name == layer_name:
bottom = False
bottom_model = Model(bottom_input, bottom_output)
top_model = Model(top_input, top_output)
return bottom_model, top_model
bottom_model, top_model = get_bottom_top_model(model, "dense_1")
Layer_name is just the name of the layer that you want to split at.
Is there a layer in Caffe that can get an arbitrary sub block in a blob?
AFAIK there is not a completely general slicing layer in caffe.
If you want to extract sub-blocks that are specific channels of a blob, you can use "Slice" layer.
Depending on your requirements, you might find "Crop" layer sufficient for your needs.
If you need more flexible access to sub-blocks, you might want to consider using a "Python" layer.
An example using "Crop" layer
As you pointed out "Crop" layer expects two "bottom"s but since the second one is only used for reference shape, you can produce it using "DummyData" layer.
Suppose you want to select x[:,:,3:20,5:40], this is a 17x35 crop
layer {
name: "ref_shape_17x35"
type: "DummyData"
top: "ref_shape_17x35"
dummy_data_param { shape { dim: 1 dim: 1 dim: 17 dim: 35 } }
}
layer {
name: "crop_x"
type: "Crop"
bottom: "x"
bottom: "ref_shape_17x35"
top: "crop_x"
crop_param {
axis: 2 # do not crop the first two dimensions
offset: 3
offset: 5
}
}
I did not try it myself, but it should work (let me know if it doesn't).
I have a network. In one place I want to use concat. As on this picture.
Unfortunately, the network doesn't train. To understand why I want to change weights in concat. Meaning that all values from FC4096 will get 1 and all values from FC16000 will get 0 at the beginning.
I know that FC4096 will get me 57% accuracy, so with learning rate 10^-6 I will understand why after concatenation layers didn't learn.
The question is, how can I set all values from FC4096 to 1 and all values from FC16000 to 0?
You can add a "Scale" layer on top of FC16000 and init it to 0:
layer {
name: "scale16000"
type: "Scale"
bottom: "fc16000"
top: "fc16000" # not 100% sure this layer can work in-place, worth trying though.
scale_param {
bias_term: false
filler: { type: "constant" value: 0 }
}
param { lr_mult: 0 decay_mult: 0 } # set mult to non zero if you want to train this scale
}
Now I'm writing the weight filler layer by layer, like
layer {
name: "Convolution1"
type: "Convolution"
bottom: "data"
top: "Convolution1"
convolution_param {
num_output: 20
kernel_size: 5
weight_filler {
type: "xavier"
}
}
}
How can I set a global weight filler type?
Thanks.
It seems currently there's no other way of doing it. In the caffe.proto file, the NetParameter is defined as follows, where there's no such option as default_weight_filler or so.
message NetParameter {
optional string name = 1; // consider giving the network a name
// DEPRECATED. See InputParameter. The input blobs to the network.
repeated string input = 3;
// DEPRECATED. See InputParameter. The shape of the input blobs.
repeated BlobShape input_shape = 8;
// 4D input dimensions -- deprecated. Use "input_shape" instead.
// If specified, for each input blob there should be four
// values specifying the num, channels, height and width of the input blob.
// Thus, there should be a total of (4 * #input) numbers.
repeated int32 input_dim = 4;
// Whether the network will force every layer to carry out backward operation.
// If set False, then whether to carry out backward is determined
// automatically according to the net structure and learning rates.
optional bool force_backward = 5 [default = false];
// The current "state" of the network, including the phase, level, and stage.
// Some layers may be included/excluded depending on this state and the states
// specified in the layers' include and exclude fields.
optional NetState state = 6;
// Print debugging information about results while running Net::Forward,
// Net::Backward, and Net::Update.
optional bool debug_info = 7 [default = false];
// The layers that make up the net. Each of their configurations, including
// connectivity and behavior, is specified as a LayerParameter.
repeated LayerParameter layer = 100; // ID 100 so layers are printed last.
// DEPRECATED: use 'layer' instead.
repeated V1LayerParameter layers = 2;
}