Fully Connected Layer Followed by LSTM Layer - machine-learning

How do i combine a Tensorflow Fully Connected Layer, which is then followed by a LSTM Layer. My goal is to feed data of batch_size=batch_size, sequence length seq_length, and dimension 1. Target is 21 dim one_hot vector.
Here is the code i tried. It throws an error Shape must be rank 2 but is rank 3 for W_first line. What am i doing wrong
data = tf.placeholder(tf.float32, [None, 20,1]) #Number of examples,number of input, dimension of each input
target = tf.placeholder(tf.float32, [None, 21])
W_first=tf.Variable(tf.random_normal([10,num_hidden]))
out_1= tf.matmul(data,W_first)
cell = tf.nn.rnn_cell.LSTMCell(num_hidden,state_is_tuple=True)
val, _ = tf.nn.dynamic_rnn(cell, out_1, dtype=tf.float32)
Thanks in advance!

Related

How to reshape Data for LSTM Time Series Prediction?

values = lstm_df_w.values
n_train_weeks = len(train_set_weekly_multi)
train = values[:n_train_weeks]
test = values[n_train_weeks:,:]
# split into input and outputs
train_X, train_y = train[:, :-1], train[:, -1]
test_X, test_y = test[:, :-1], test[:, -1]
# reshape input to be 3D [samples, timesteps, features]
train_X = train_X.reshape((train_X.shape[0], 1, train_X.shape[1]))
test_X = test_X.reshape((test_X.shape[0], 1, test_X.shape[1]))
Hello Everyone,
at the moment i am trying to do a multivaraite Time Series Forecasting Projekt.
I managed to build a Model and to the forecast.
However it is only using 1 Timestemp of the past. But i want to try different past Time Stamps for the prediction.
If i change the value to 2 i get this error message:
cannot reshape array of size 2322 into shape (258,2,9)
Does anyone now how to deal with this problem?
Picture of the DF:
Picture of Data

Using softmax in Neural Networks to decide label of input

I am using the keras model with the following layers to predict a label of input (out of 4 labels)
embedding_layer = keras.layers.Embedding(MAX_NB_WORDS,
EMBEDDING_DIM,
weights=[embedding_matrix],
input_length=MAX_SEQUENCE_LENGTH,
trainable=False)
sequence_input = keras.layers.Input(shape = (MAX_SEQUENCE_LENGTH,),
dtype = 'int32')
embedded_sequences = embedding_layer(sequence_input)
hidden_layer = keras.layers.Dense(50, activation='relu')(embedded_sequences)
flat = keras.layers.Flatten()(hidden_layer)
preds = keras.layers.Dense(4, activation='softmax')(flat)
model = keras.models.Model(sequence_input, preds)
model.compile(loss='categorical_crossentropy', optimizer='rmsprop', metrics=['acc'])
model.fit(X_train, Y_train, batch_size=32, epochs=100)
However, the softmax function returns a number of outputs of 4 (because I have 4 labels)
When I'm using the predict function to get the predicted Y using the same model, I am getting an array of 4 for each X rather than one single label deciding the label for the input.
model.predict(X_test, batch_size = None, verbose = 0, steps = None)
How do I make the output layer of the keras model, or the model.predict function, decide on one single label, rather than output weights for each label?
The following is a common function to sample from a probability vector
def sample(preds, temperature=1.0):
# helper function to sample an index from a probability array
preds = np.asarray(preds).astype('float64')
preds = np.log(preds) / temperature
exp_preds = np.exp(preds)
preds = exp_preds / np.sum(exp_preds)
probas = np.random.multinomial(1, preds, 1)
return np.argmax(probas)
Taken from here.
The temperature parameter decides how much the differences between the probability weights are weightd. A temperature of 1 is considering each weight "as it is", a temperature larger than 1 reduces the differences between the weights, a temperature smaller than 1 increases them.
Here an example using a probability vector on 3 labels:
p = np.array([0.1, 0.7, 0.2]) # The first label has a probability of 10% of being chosen, the second 70%, the third 20%
print(sample(p, 1)) # sample using the input probabilities, unchanged
print(sample(p, 0.1)) # the new vector of probabilities from which to sample is [ 3.54012033e-09, 9.99996371e-01, 3.62508322e-06]
print(sample(p, 10)) # the new vector of probabilities from which to sample is [ 0.30426696, 0.36962778, 0.32610526]
To see the new vector make sample return preds.

Issue in LSTM Input Dimensions in Keras

I am trying to implement a multi-input LSTM model using keras. The code is as follows:
data_1 -> shape (1150,50)
data_2 -> shape (1150,50)
y_train -> shape (1150,50)
input_1 = Input(shape=data_1.shape)
LSTM_1 = LSTM(100)(input_1)
input_2 = Input(shape=data_2.shape)
LSTM_2 = LSTM(100)(input_2)
concat = Concatenate(axis=-1)
x = concat([LSTM_1, LSTM_2])
dense_layer = Dense(1, activation='sigmoid')(x)
model = keras.models.Model(inputs=[input_1, input_2], outputs=[dense_layer])
model.compile(loss='binary_crossentropy',
optimizer='adam',
metrics=['acc'])
model.fit([data_1, data_2], y_train, epochs=10)
When I run this code, I get a ValueError:
ValueError: Error when checking model input: expected input_1 to have 3 dimensions, but got array with shape (1150, 50)
Do anyone have any solution to this problem?
Use data1 = np.expand_dims(data1, axis=2), before you define the model. LSTM expects inputs with dimensions (batch_size, timesteps, features), so, in your case, I guessing you have 1 feature, 50 time steps and 1150 samples, you need to add a dimension at the end of your vector.
This need to be done before you define the model otherwise when you set input_1 = Input(shape=data_1.shape) you are telling keras that your input has 1150 timesteps and 50 features,so it will expect inputs of shape (None, 1150, 50) (the non stands for "any dimension will be accepted").
The same holds for input_2
Hope this helps

How can I get biases from a trained model in Keras?

I have built a simple neural network,
model = Sequential()
model.add(Dense(20, input_dim=5, activation='sigmoid'))
model.add(Dense(1, activation='sigmoid'))
and I would get its weights by:
summary = model.summary()
W_Input_Hidden = model.layers[0].get_weights()[0]
W_Output_Hidden = model.layers[1].get_weights()[0]
print(summary)
print('INPUT-HIDDEN LAYER WEIGHTS:')
print(W_Input_Hidden)
print('HIDDEN-OUTPUT LAYER WEIGHTS:')
print(W_Output_Hidden)
but, in this way, I only get the weights matrices (5x20 , 1x20) without the biases. How can I get the biases values?
Quite simple, its just the second element in the array returned by get_weights() (For Dense layers):
B_Input_Hidden = model.layers[0].get_weights()[1]
B_Output_Hidden = model.layers[1].get_weights()[1]
Here's a complete working example (implemented with TensorFlow 2 and Keras).
import tensorflow as tf
import numpy as np
def get_model():
inp = tf.keras.layers.Input(shape=(1,))
# Use the parameter bias_initializer='random_uniform'
# in case you want the initial biases different than zero.
x = tf.keras.layers.Dense(8)(inp)
out = tf.keras.layers.Dense(1)(x)
model = tf.keras.models.Model(inputs=inp, outputs=out)
return model
def main():
model = get_model()
model.compile(loss="mse")
weights = model.layers[1].get_weights()[0]
biases = model.layers[1].get_weights()[1]
print("initial weights =", weights)
print("initial biases =", biases)
X = np.random.randint(-10, 11, size=(1000, 1))
y = np.random.randint(0, 2, size=(1000, 1))
model.fit(X, y)
weights = model.layers[1].get_weights()[0]
biases = model.layers[1].get_weights()[1]
print("learned weights =", weights)
# Biases are similar because they are all initialized with zeros (by default).
print("learned biases =", biases)
if __name__ == '__main__':
main()
YOu can view and output biases and weights using the following code:
for layer in model.layers:
g=layer.get_config()
h=layer.get_weights()
print (g)
print (h)
if you're looking for weights and bias from the validation dataset, you need to do model.predict on each vector from the dataset.
for i in range(len(valData)):
ValResults = model.predict(valData[i])
B_Input_Hidden = model.layers[0].get_weights()[1]
B_Output_Hidden = model.layers[1].get_weights()[1]

keras model fit_generator ValueError: Error when checking model target: expected cropping2d_4 to have 4 dimensions, but got array with shape (32, 1)

I'm trying to use keras model.fit_generator() to fit a model, below is my definition of the generator:
from sklearn.utils import shuffle
IMG_PATH_PREFIX = "./data/IMG/"
def generator(samples, batch_size=64):
num_samples = len(samples)
while 1: # Loop forever so the generator never terminates
shuffle(samples)
for offset in range(0, num_samples, batch_size):
batch_samples = samples[offset:offset+batch_size]
images = []
angles = []
for batch_sample in batch_samples:
name = IMG_PATH_PREFIX + batch_sample[0].split('/')[-1]
center_image = cv2.imread(name)
center_angle = float(batch_sample[3])
images.append(center_image)
angles.append(center_angle)
X_train = np.array(images)
y_train = np.array(angles)
#X_train = np.expand_dims(X_train, axis=0)
#y_train = np.expand_dims(y_train, axis=1)
print("X_train shape: ", X_train.shape, " y_train shape:", y_train.shape)
#print("X train: ", X_train)
yield X_train, y_train
train_generator = generator(train_samples, batch_size = 32)
validation_generator = generator(validation_samples, batch_size = 32)
Here the output shape is:
X_train shape: (32, 160, 320, 3) y_train shape: (32,)
The model fit code is:
model = Sequential()
#cropping layer
model.add(Cropping2D(cropping=((50,20), (1,1)), input_shape=(160,320,3), dim_ordering='tf'))
model.compile(loss = "mse", optimizer="adam")
model.fit_generator(train_generator, samples_per_epoch= len(train_samples), validation_data=validation_generator, nb_val_samples=len(validation_samples), nb_epoch=3)
Then I get the error message:
ValueError: Error when checking model target: expected cropping2d_6 to have 4 dimensions, but got array with shape (32, 1)
Could someone help let me know what's the issue?
The big question here is : do you know what you are trying to do ?
1) If you read here, the input is a 4D tensor and the output is ALSO a 4D tensor. Your target is a 2D tensor of shape (batch_size,1). So of course, when keras tries to compute the error between the output which has 3D (without batch dimension) and the target which has 1D (without batch dimension), it can not make sense out of that. Outputs and targets must have the same dimensions.
2) Do you know what cropping2D is actually doing ? It is cropping your images... So removing values at the beginning and end of your cropping dimensions. In your case you are outputing images of shape (90, 218, 3). This is not a prediction, there is no weight to train on this layer so no reason to fit the "model". Your model is just cropping images. No training needed for that.

Resources