I have a network. In one place I want to use concat. As on this picture.
Unfortunately, the network doesn't train. To understand why I want to change weights in concat. Meaning that all values from FC4096 will get 1 and all values from FC16000 will get 0 at the beginning.
I know that FC4096 will get me 57% accuracy, so with learning rate 10^-6 I will understand why after concatenation layers didn't learn.
The question is, how can I set all values from FC4096 to 1 and all values from FC16000 to 0?
You can add a "Scale" layer on top of FC16000 and init it to 0:
layer {
name: "scale16000"
type: "Scale"
bottom: "fc16000"
top: "fc16000" # not 100% sure this layer can work in-place, worth trying though.
scale_param {
bias_term: false
filler: { type: "constant" value: 0 }
}
param { lr_mult: 0 decay_mult: 0 } # set mult to non zero if you want to train this scale
}
Related
I managed to create a CoreML 2.0 model with flexible input/output shape sizes:
I can't figure out how to set the size in my Xcode project, however. If I set the input pixel buffer size 2048x2048, the output pixel buffer is still 1536x1536. If I set it to 768x768, the resulting pixel buffer is still 1536x1536 - but is blank outside the region of 768x768.
I examined the automatically generated Swift model class and don't see any clues there.
I can't find a single example anywhere showing how to use the "Flexibility" sizes.
In the WWDC 2018 Session 708 "What's New in Core ML", Part 1 it states:
This means that now you have to ship a single model. You don't have to have any redundant code. And if you need to switch between standard definition and high definition, you can do it much faster because we don't need to reload the model from scratch; we just need to resize it. You have two options to specify the flexibility of the model. You can define a range for its dimension, so you can define a minimal width and height and the maximum width and height. And then at inference pick any value in between. But there is also another way. You can enumerate all the shapes that you are going to use. For example, all different aspect ratios, all different resolutions, and this is better for performance. Core ML knows more about your use case earlier, so it can -- it has the opportunities of performing more optimizations.
They say "we just need to resize it". It so frustrating because they don't tell you how to just resize it! They also say "And then at inference pick any value in between" but offer no clue how to pick the value in between!
Here is how I added the flexible shape sizes:
import coremltools
from coremltools.models.neural_network import flexible_shape_utils
spec = coremltools.utils.load_spec('mymodel_fxedShape.mlmodel')
img_size_ranges = flexible_shape_utils.NeuralNetworkImageSizeRange()
img_size_ranges.add_height_range(640, 2048)
img_size_ranges.add_width_range(640, 2048)
flexible_shape_utils.update_image_size_range(spec, feature_name='inputImage', size_range=img_size_ranges)
flexible_shape_utils.update_image_size_range(spec, feature_name='outputImage', size_range=img_size_ranges)
coremltools.utils.save_spec(spec, 'myModel.mlmodel')
Here is the description of the model:
description {
input {
name: "inputImage"
shortDescription: "Image to stylize"
type {
imageType {
width: 1536
height: 1536
colorSpace: BGR
imageSizeRange {
widthRange {
lowerBound: 640
upperBound: 2048
}
heightRange {
lowerBound: 640
upperBound: 2048
}
}
}
}
}
output {
name: "outputImage"
shortDescription: "Stylized image"
type {
imageType {
width: 1536
height: 1536
colorSpace: BGR
imageSizeRange {
widthRange {
lowerBound: 640
upperBound: 2048
}
heightRange {
lowerBound: 640
upperBound: 2048
}
}
}
}
}
}
There are two layers using "outputShape":
layers {
name: "SpatialFullConvolution_63"
input: "Sequential_53"
output: "SpatialFullConvolution_63_output"
convolution {
outputChannels: 16
kernelChannels: 32
nGroups: 1
kernelSize: 3
kernelSize: 3
stride: 2
stride: 2
dilationFactor: 1
dilationFactor: 1
valid {
paddingAmounts {
borderAmounts {
}
borderAmounts {
}
}
}
isDeconvolution: true
hasBias: true
weights {
}
bias {
}
outputShape: 770
outputShape: 770
}
}
...relu layer...
layers {
name: "SpatialFullConvolution_67"
input: "ReLU_66"
output: "SpatialFullConvolution_67_output"
convolution {
outputChannels: 8
kernelChannels: 16
nGroups: 1
kernelSize: 3
kernelSize: 3
stride: 2
stride: 2
dilationFactor: 1
dilationFactor: 1
valid {
paddingAmounts {
borderAmounts {
}
borderAmounts {
}
}
}
isDeconvolution: true
hasBias: true
weights {
}
bias {
}
outputShape: 1538
outputShape: 1538
}
}
I am now trying to figure out how to remove the outputShape from those two layers.
>>> layer = spec.neuralNetwork.layers[49]
>>> layer.convolution.outputShape
[1538L, 1538L]
I tried setting it to []:
layer.convolution.outputShape = []
To a Shape:
layer.convolution.outputShape = flexible_shape_utils.Shape(())
Whatever I try, I get the error:
TypeError: Can't set composite field
Do I have to create a new layer and then link it to the layer that is outputting to it and the layer it is outputting to?
The issue in this case was that there were layers present in the model that used a fixed shape for their outputShapes. For example:
>>> layer = spec.neuralNetwork.layers[49]
>>> layer.convolution.outputShape
[1538L, 1538L]
The model in question was indeed fully convolutional, so before converting to CoreML, it worked with any input and output shapes.
I was able to delete the fixed outputShape with this command:
layer = spec.neuralNetwork.layers[49]
del layer.convolution.outputShape[:]
After doing that, the model worked with flexible input and output shapes.
All credit for this answer goes to Matthijs Hollemans.
Is there a layer in Caffe that can get an arbitrary sub block in a blob?
AFAIK there is not a completely general slicing layer in caffe.
If you want to extract sub-blocks that are specific channels of a blob, you can use "Slice" layer.
Depending on your requirements, you might find "Crop" layer sufficient for your needs.
If you need more flexible access to sub-blocks, you might want to consider using a "Python" layer.
An example using "Crop" layer
As you pointed out "Crop" layer expects two "bottom"s but since the second one is only used for reference shape, you can produce it using "DummyData" layer.
Suppose you want to select x[:,:,3:20,5:40], this is a 17x35 crop
layer {
name: "ref_shape_17x35"
type: "DummyData"
top: "ref_shape_17x35"
dummy_data_param { shape { dim: 1 dim: 1 dim: 17 dim: 35 } }
}
layer {
name: "crop_x"
type: "Crop"
bottom: "x"
bottom: "ref_shape_17x35"
top: "crop_x"
crop_param {
axis: 2 # do not crop the first two dimensions
offset: 3
offset: 5
}
}
I did not try it myself, but it should work (let me know if it doesn't).
The benefit using in_place=True for a layer is for saving memory. However, I found some paper like ResNet that use in_place=False for the BatchNorm layer. My question is when do we use in_place=True/False for the BatchNorm layer. Does it has any effect on performance? Thanks all
layer {
name: "bnorm1"
type: "BatchNorm"
bottom: "conv1_bn"
top: "conv1_bn"
batch_norm_param {
use_global_stats: false
}
}
The related code to make above prototxt is
n.bnorm1 = L.BatchNorm(n.conv1_bn, batch_norm_param=dict(use_global_stats=False), in_place=True)
n.scale1 = L.Scale(n.bnorm1, in_place=True, bias_term=True,filler=dict(value=1), bias_filler=dict(value=0))
I'm looking to achieve an effect similar to that in this example.
However, the above example uses a gradient that is relative to the visible plot. So the 0 stop color is used at the peak of the visible plot (whatever the actual value of the peak), and the 1 stop color is used at the base.
Say I have data that can range from 0 to 100 (e.g. a percentage). I want to be able to specify that "dark" is always used for a value of 100, while "light" is always used for a value of 0.
So, if my visible data only ranges from 0 to 20, then this is all filled with a light-ish gradient colour (at the lower end of the gradient spectrum)... I don't want it to be filled from light to fully dark.
Any way to achieve this?
To achieve such gradient, you need to do a few things.
Gradient units should be switched to userSpaceOnUse.
x1, y1, x2, y2 should point to the points where the plot area starts and ends.
You do not know what would be the plot area, so you have to set the gradient on load event.
function() {
this.series[0].update({
fillColor: {
linearGradient: [0, 0, 0, this.plotHeight]
}
});
}
example: http://jsfiddle.net/qhuee8uf/1/
Now, the gradient has fixed x/y attributes, what means that the gradient will not be responsive. To have a gradient which will work with a resizing chart, you should recalculate gradient on redraw event.
Also, I have found that updating gradient in the series does not update the gradient but creates a new one - which might be not be the best behaviour to have.
Instead, you can modify the gradient attributes directly
function adjustGradient() {
document.getElementsByTagName('linearGradient')[0].setAttributeNS(null, 'y2', this.plotHeight);
}
chart: {
events: {
load: adjustGradient,
redraw: adjustGradient
}
},
example: http://jsfiddle.net/qhuee8uf/
Now I'm writing the weight filler layer by layer, like
layer {
name: "Convolution1"
type: "Convolution"
bottom: "data"
top: "Convolution1"
convolution_param {
num_output: 20
kernel_size: 5
weight_filler {
type: "xavier"
}
}
}
How can I set a global weight filler type?
Thanks.
It seems currently there's no other way of doing it. In the caffe.proto file, the NetParameter is defined as follows, where there's no such option as default_weight_filler or so.
message NetParameter {
optional string name = 1; // consider giving the network a name
// DEPRECATED. See InputParameter. The input blobs to the network.
repeated string input = 3;
// DEPRECATED. See InputParameter. The shape of the input blobs.
repeated BlobShape input_shape = 8;
// 4D input dimensions -- deprecated. Use "input_shape" instead.
// If specified, for each input blob there should be four
// values specifying the num, channels, height and width of the input blob.
// Thus, there should be a total of (4 * #input) numbers.
repeated int32 input_dim = 4;
// Whether the network will force every layer to carry out backward operation.
// If set False, then whether to carry out backward is determined
// automatically according to the net structure and learning rates.
optional bool force_backward = 5 [default = false];
// The current "state" of the network, including the phase, level, and stage.
// Some layers may be included/excluded depending on this state and the states
// specified in the layers' include and exclude fields.
optional NetState state = 6;
// Print debugging information about results while running Net::Forward,
// Net::Backward, and Net::Update.
optional bool debug_info = 7 [default = false];
// The layers that make up the net. Each of their configurations, including
// connectivity and behavior, is specified as a LayerParameter.
repeated LayerParameter layer = 100; // ID 100 so layers are printed last.
// DEPRECATED: use 'layer' instead.
repeated V1LayerParameter layers = 2;
}