i used to try normal dropout and it always gives better result
so this time I wanted to try the Monte Carlo method to see how it works but it doesn't change anything at any rate; it all seems like before. this is my code:
with keras.backend.learning_phase_scope(1):
ypr=np.stack([model.predict(xscale)
for sample in range(100)])
y=ypr.mean(axis=0)
do i miss something?
thanks
I have encountered the same problem. An alternative is to subclass the Dropout layer and override the call() method to force its training argument to True:
class MonteCarloDropout(Dropout):
def call(self, inputs):
return super().call(inputs, training=True)
Replace the Dropout layers in your model with MonteCarloDropout layers and it will work as you expect.
Related
Is it possible to use dropout at train and test phase in Keras?
Like described here:
https://github.com/soumith/ganhacks#17-use-dropouts-in-g-in-both-train-and-test-phase
Sure, you can set training argument to True when calling the Dropout layer. In this way, dropout would be applied in both training and test phases:
drp_output = Dropout(rate)(inputs, training=True) # dropout would be active in train and test phases
Both answers leave me slightly confused. More simply, you may find yourself doing something like this:
model = Model(...)
...
model.add(Dropout(0.5))
...
model.fit(...) # invokes Dropout(training=True)
...
model.evaluate(...) # invokes Dropout(training=False)
That is, when you define your model, you add Dropout layers with the dropout rate you want during training. The rate is not visibly varied between test and training; rather, it is declared once as a fixed value, then (invisibly) switched on/off according to the training parameter the layer is invoked with. See keras.Model.
I'm working with RetinaNet NN model for object detection and I faced with over fitting problem.
One of the solutions is adding "Dropout".
I'm Using the keras code Here
I want to Add Dropout to the last layers but I don't know how to add.
Can anyone help which file should I change?and how?
After a while, I tried many solutions but non of them didn't say how to add exactly, so I tried and then found how to add, So decided to answer it myself!
Just need to add a line like this:
outputs = keras.layers.SpatialDropout1D (rate=dropout_rate) (outputs)
You can use another layer dropout type like :
SpatialDropout2D and more.
You could try to store the fully connected layer into a variable like:
fc1 = model.layers[-3]
fc2 = model.layers[-2]
predictions = model.layers[-1]
Then create your dropout layer and reconnect them all to build a new Model as shown in this post :Add dropout layers between pretrained dense layers in keras
Hope this helps.
I am trying to classify different ECG signals. I am using Keras' Conv1D, but am not getting any good results.
I have tried changing the number of layers, window size, etc, but every time I run this I get predictions all of the same class (the classes are 0,1,2, so I get a prediction output of something like [1,1,1,1,1,1,1,1,1,1,1,1,1,1], but the class changes each time I run the script).
The ECG signals are in 1000 point numpy arrays.
Are there any glaringly obvious things I am doing wrong here? I was thinking it would've worked great to use a few layers to just classify into 3 different ECG signals.
#arrange and randomize data
y1=[[0]]*len(lead1)
y2=[[1]]*len(lead2)
y3=[[2]]*len(lead3)
y=np.concatenate((y1,y2,y3))
data=np.concatenate((lead1,lead2,lead3))
data = keras.utils.normalize(data)
data=np.concatenate((data,y),axis=1)
data=np.random.permutation((data))
print(data)
#separate data and create categories
Xtrain=data[0:130,0:-1]
Xtrain=np.reshape(Xtrain,(len(Xtrain),1000,1))
Xpred=data[130:,0:-1]
Xpred=np.reshape(Xpred,(len(Xpred),1000,1))
Ytrain=data[0:130,-1]
Yt=to_categorical(Ytrain)
Ypred=data[130:,-1]
Yp=to_categorical(Ypred)
#create CNN model
model = Sequential()
model.add(Conv1D(20,20,activation='relu',input_shape=(1000,1)))
model.add(MaxPooling1D(3))
model.add(Conv1D(20,10,activation='relu'))
model.add(MaxPooling1D(3))
model.add(Conv1D(20,10,activation='relu'))
model.add(GlobalAveragePooling1D())
model.add(Dense(3,activation='relu',use_bias=False))
model.compile(optimizer='adam', loss='categorical_crossentropy',metrics=['accuracy'])
model.fit(Xtrain,Yt)
#test model
print(model.evaluate(Xpred,Yp))
print(model.predict_classes(Xpred,verbose=1))
Are there any glaringly obvious things I am doing wrong here?
Indeed there is: the output you report is not surprising, given that you are currently using the ReLU as activation for your last layer, which does not make any sense.
In multi-class settings, such as yours, the activation of the last layer must be the softmax, and certainly not the ReLU; change your last layer to:
model.add(Dense(3, activation='softmax'))
Not quite sure why you ask for use_bias=False, but you can try both with and without it and experiment...
In keras' documentation there is no information regarding how dropout is actually implemented for LSTM layers.
However, there is a link to the paper "A Theoretically Grounded Application of Dropout in Recurrent Neural Networks", which led me to believe that dropout is implemented as described in said paper.
That is, for each time-step in the time-series the layer is processing, the same dropout mask is used.
Looking at the source code, it seems to me that LSTMCell.call gets called iteratively, once for every time-step in the time-series, and generates a new dropout mask each time it is called.
My question is:
Either I misinterpreted keras' code, or the reference to the paper in keras' documentation is misleading. Which is it?
Both the paper and the code are consistent. You have understood correctly but interpreted the code a bit wrong.
There is a check before initialising dropout_mask, self._dropout_mask is None
So LSTMCell.call gets called iteratively, once for every time-step in the time-series, but only in the first call a new dropout mask is generated.
if 0 < self.dropout < 1 and self._dropout_mask is None:
self._dropout_mask = _generate_dropout_mask(
K.ones_like(inputs),
self.dropout,
training=training,
count=4)
if (0 < self.recurrent_dropout < 1 and
self._recurrent_dropout_mask is None):
self._recurrent_dropout_mask = _generate_dropout_mask(
K.ones_like(states[0]),
self.recurrent_dropout,
training=training,
count=4)
Hope that clarifies your doubt.
I'm trying to program a neural network with backpropagation in python.
Usually converges to 1. To the left of the image there are some delta values. They are very small, should they be larger? Do you know a reason why this converging could happen?
sometimes it goes up in the direction of the point and then goes down again
here is the complete code:
http://pastebin.com/9BiwhWrD the backpropagation code starts at line 146
(the root stuff in line 165 does nothing. was just trying out some ideas)
Any ideas of what could be wrong? Have you ever seen a behaviour like this?
Thanks you very much.
The reason why this happened is, because the input data was too large. The activation sigmoid function converged to f(x)=1 for x -> inf. I had to normalize the data
e.g.:
a = np.array([1,2,3,4,5])
a /= a.max()
or prevent generating unnormalized data at all.
Also, the interims value was updated BEFORE the sigmoid was applied. But the derivation of sigmoid looks like this: y'(x) = y(x)-(1-y(x)). In my case it was just: y'(x) = x-(1-x)
There were also errors in how i updated the weights after calculating the deltas. I rewrote the whole loop using a tutorial for neural networks with python and then it worked.
It still does not support bias but it can do classification. For regression it's not precise enough, but i guess this has to do with the missing bias.
Here is the code:
http://pastebin.com/hRCKe1dK
Someone suggested that i should put my training-data into a neural-network framework and see if it works. It didn't. So it was kindof clear that it had to to with it and so i had to the idea that it should be between -1 and 1.