Related
I'm working with MxNet and I'm figuring out the Seq2Seq model. Let’s suppose that every batch will handle 32 sequences and every sequence will be of length 20 (timesteps). In order to create the architecture to work with seq2seq models we are going to split every sequence into two parts. The methods of splitting is very arbitrary but let’s suppose we divide in half the sequence. The first part will be named ‘encoder input’ and will, indeed, be the input to the encoder which will consist in a sequence of 10 (timesteps), clearly this input consist in N numbers of variable of length 10. Therefore, we’ll have x1, … , x10 for every encoding input sequence multiplied by the number of features which will result into the feature vector of encoding inputs Xt. Now, since the decoder output will be the second half of the sequence, what should be the decoder input? I'm setting the decoder input as the encoder input and tha model is working fairly good. That's the forward function:
def forward(self, encoder_input, *args):
state= self.encoder.begin_state(batch_size=encoder_input.shape[0], ctx=mx.cpu())
encoder_output, encoder_state= self.encoder(encoder_input, state)
decoder_output, decoder_state= self.decoder(encoder_input, encoder_state)
output= self.dense(decoder_output)
return output
Is there some error with using encoder input as decoder input? I've seen some example in Keras where they initialize decoder input as an np.array with the shape of the decoder output. I've tried to set decoder input like an array of zeros but the results (in terms of accuracy) decay really badly.
I've found on 'Hands on Machine Learning':
In other words,
the decoder is given as input the word that it should have output at the
previous step (regardless of what it actually output). For the very first
word, it is given the start-of-sequence (SOS) token. The decoder is
expected to end the sentence with an end-of-sequence (EOS) token.
Therefore, I suppose that if the encoder input will be composed by the first sequence of n observation for the z features, no matter what is the encoder output, we should feed the decoder with the encoder states and the decoder input which is the expected output of the encoder or in other words the sequence of the first n observation of the label. Despite all, in my analysis with python there are no evidence of better results. Maybe, feeding the decoder with only the encoder label is better when we got a lot of features.
I am trying to merge output from two models and give them as input to the third model using keras sequential model.
Model1 :
inputs1 = Input(shape=(750,))
x = Dense(500, activation='relu')(inputs1)
x = Dense(100, activation='relu')(x)
Model1 :
inputs2 = Input(shape=(750,))
y = Dense(500, activation='relu')(inputs2)
y = Dense(100, activation='relu')(y)
Model3 :
merged = Concatenate([x, y])
final_model = Sequential()
final_model.add(merged)
final_model.add(Dense(100, activation='relu'))
final_model.add(Dense(3, activation='softmax'))
Till here, my understanding is that, output from two models as x and y are merged and given as input to the third model. But when I fit this all like,
module3.compile(optimizer='rmsprop', loss='categorical_crossentropy', metrics=['accuracy'])
module3.fit([in1, in2], np_res_array)
in1 and in2 are two numpy ndarray of dimention 10000*750 which contains my training data and np_res_array is the corresponding target. This gives me error as 'list' object has no attribute 'shape' As far as know, this is how we give multiple inputs to a model, but what is this error? How do I resolve it?
You can't do this using Sequential API. That's because of two reasons:
Sequential models, as their name suggests, are a sequence of layers where each layer is connected directly to its previous layer and therefore they cannot have branches (e.g. merge layers, multiple input/output layers, skip connections, etc.).
The add() method of Sequential API accepts a Layer instance as its argument and not a Tensor instance. In your example merged is a Tensor (i.e. concatenation layer's output).
Further, the correct way of using Concatenate layer is like this:
merged = Concatenate()([x, y])
However, you can also use concatenate (note the lowercase "c"), its equivalent functional interface, like this:
merged = concatenate([x, y])
Finally, to be able to construct that third model you also need to use the functional API.
I have a similar open question here on Cross Validated (though not implementation focused, which I intend this question to be, so I think they are both valid).
I'm working on a project that uses sensors to monitor a persons GPS location. The coordinates will then be converted to a simple-grid representation. What I want to try and do is after recording a users routes, train a neural network to predict the next coordinates, i.e. take the example below where a user repeats only two routes over time, Home->A and Home->B.
I want to train an RNN/LSTM with sequences of varying lengths e.g. (14,3), (13,3), (12,3), (11,3), (10,3), (9,3), (8,3), (7,3), (6,3), (5,3), (4,3), (3,3), (2,3), (1,3) and then also predict with sequences of varying lengths e.g. for this example route if I called
route = [(14,3), (13,3), (12,3), (11,3), (10,3)] //pseudocode
pred = model.predict(route)
pred should give me (9,3) (or ideally even a longer prediction e.g. ((9,3), (8,3), (7,3), (6,3), (5,3), (4,3), (3,3), (2,3), (1,3))
How do I feed such training sequences to the init and forward operations identified below?
self.rnn = nn.RNN(input_size, hidden_dim, n_layers, batch_first=True)
out, hidden = self.rnn(x, hidden)
Also, should the entire route be a tensor or each set of coordinates within the route a tensor?
I'm not very experienced with RNNs, but I'll give it a try.
A few things to pay attention to before we start:
1. Your data is not normalized.
2. The output prediction you want (even after normalization) is not bounded to [-1, 1] range and therefore you cannot have tanh or ReLU activations acting on the output predictions.
To address your problem, I propose a recurrent net that given a current state (2D coordinate) predicts the next state (2D coordinates). Note that since this is a recurrent net, there is also a hidden state associated with each location. At first, the hidden state is zero, but as the net sees more steps, it updates its hidden state.
I propose a simple net to address your problem. It has a single RNN layer with 8 hidden states, and a fully connected layer on to to output the prediction.
class MyRnn(nn.Module):
def __init__(self, in_d=2, out_d=2, hidden_d=8, num_hidden=1):
super(MyRnn, self).__init__()
self.rnn = nn.RNN(input_size=in_d, hidden_size=hidden_d, num_layers=num_hidden)
self.fc = nn.Linear(hidden_d, out_d)
def forward(self, x, h0):
r, h = self.rnn(x, h0)
y = self.fc(r) # no activation on the output
return y, h
You can use your two sequences as training data, each sequence is a tensor of shape Tx1x2 where T is the sequence length, and each entry is two dimensional (x-y).
To predict (during training):
rnn = MyRnn()
pred, out_h = rnn(seq[:-1, ...], torch.zeros(1, 1, 8)) # given time t predict t+1
err = criterion(pred, seq[1:, ...]) # compare prediction to t+1
Once the model is trained, you can show it first k steps and continue to predict the next steps:
rnn.eval()
with torch.no_grad():
pred, h = rnn(s[:k,...], torch.zeros(1, 1, 8, dtype=torch.float))
# pred[-1, ...] is the predicted next step
prev = pred[-1:, ...]
for j in range(k+1, s.shape[0]):
pred, h = rnn(prev, h) # note how we keep track of the hidden state of the model. it is no longer init to zero.
prev = pred
I put everything together in a colab notebook so you can play with it.
For simplicity, I ignored the data normalization here, but you can find it in the colab notebook.
What's next?
These types of predictions are prone to error accumulation. This should be addressed during training, by shifting the inputs from the ground truth "clean" sequences to the actual predicted sequences, so the model will be able to compensate for its errors.
I'm trying to make a network that outputs a depth map, and semantic segmentation data separately.
In order to train the network, I'd like to use categorical cross entropy for the segmentation branch, and mean squared error for the branch that outputs the depth map.
I couldn't find any info on implementing the two loss functions for each branches in the Keras documentation for the Functional API.
Is it possible for me to use these loss functions simultaneously during training, or would it be better for me to train the different branches separately?
From the documentation of Model.compile:
loss: String (name of objective function) or objective function. See
losses. If the model has multiple outputs, you can use a different
loss on each output by passing a dictionary or a list of losses. The
loss value that will be minimized by the model will then be the sum of
all individual losses.
If your output is named, you can use a dictionary mapping the names to the corresponding losses:
x = Input((10,))
out1 = Dense(10, activation='softmax', name='segmentation')(x)
out2 = Dense(10, name='depth')(x)
model = Model(x, [out1, out2])
model.compile(loss={'segmentation': 'categorical_crossentropy', 'depth': 'mse'},
optimizer='adam')
Otherwise, use a list of losses (in the same order as the corresponding model outputs).
x = Input((10,))
out1 = Dense(10, activation='softmax')(x)
out2 = Dense(10)(x)
model = Model(x, [out1, out2])
model.compile(loss=['categorical_crossentropy', 'mse'], optimizer='adam')
I would like to ask if would it be possible (rather if it can make any sense) to use a variational autoencoder for feature extraction. I ask because for the encoding part we sample from a distribution, and then it means that the same sample can have a different encoding (Due to the stochastic nature in the sampling process). Thanks!
Yes the feature extraction goal is the same for vae's or sparse autoencoders.
Once you have an encoder plug-in a classifier on the extracted features.
Best reggards,
Yes the output of encoder network can be used as your feature.
Just think about this: using the output of encoder network as input, the decoder network can generate you an image quite like your old image. Therefore the output of encoder network has pretty much covered most of the information in your original image. In other words, they are the most important features of your original image that distinguish it from other images.
The only thing you want to pay attention to is that variational autoencoder is a stochastic feature extractor, while usually the feature extractor is deterministic. You can either use the mean and variance as your extracted feature, or use Monte Carlo method by drawing from the Gaussian distribution defined by the mean and variance as "sampled extracted features".
Yes, you can.
I used the below code to extract the important features from my dataset.
prostate_df <- read.csv('your_data')
prostate_df <- prostate_df[,-1] # first column.
train_df<-prostate_df
outcome_name <- 'subtype' # my label column
feature_names <- setdiff(names(prostate_df), outcome_name)
library(h2o)
localH2O = h2o.init()
prostate.hex<-as.h2o(train_df, destination_frame="train.hex")
prostate.dl = h2o.deeplearning(x = feature_names,
#y="subtype",
training_frame = prostate.hex,
model_id = "AE100",
# input_dropout_ratio = 0.3, #Quite high,
#l2 = 1e-5, #Quite high
autoencoder = TRUE,
#validation_frame = prostate.hex,
#reproducible = T,seed=1,
hidden = c(1), epochs = 700,
#activation = "Tanh",
#activation ="TanhWithDropout",
activation ="Rectifier",
#activation ="RectifierWithDropout",
standardize = TRUE,
#regression_stop = -1,
#stopping_metric="MSE",
train_samples_per_iteration = 0,
variable_importances=TRUE
)
label1<-ncol(train_df)
train_supervised_features2 = h2o.deepfeatures(prostate.dl, prostate.hex, layer=1)
plotdata = as.data.frame(train_supervised_features2)
plotdata$label = as.character(as.vector(train_df[,label1]))
library(ggplot2)
qplot(DF.L1.C1, DF.L1.C2, data = plotdata, color = label, main = "Cancer Normal Pathway data ")
prostate.anon = h2o.anomaly(prostate.dl, prostate.hex, per_feature=FALSE)
head(prostate.anon)
err <- as.data.frame(prostate.anon)
h2o.scoreHistory(prostate.dl)
head(h2o.varimp(prostate.dl),10)
h2o.varimp_plot(prostate.dl)