Hello I'm using RandomizedSearchCV for hyperparameter tuning of my LSTM. The code works fine with stateful=False. However I also want so try this with stateful on but I'm not sure how to.
I arranged my data in a sliding window with shape (211845 datapoints, 4 window size, 16 features).
Following function for creating the architecture of the model.
def create_lstm(dropout_rate=0.0, neurons=32, lr=1e-3):
lstm = Sequential()
lstm.add(InputLayer((4, 16)))
lstm.add(LSTM(neurons, return_sequences=True))
lstm.add(Dropout(dropout_rate))
lstm.add(LSTM(neurons)
lstm.add(Dropout(dropout_rate))
lstm.add(Dense(neurons/4, activation='relu'))
lstm.add(Dense(1))
lstm.compile(loss='mse',
optimizer=Adam(learning_rate=lr),
metrics=['mean_squared_error']
)
return lstm
I pass the function to the wrapper
lstm_estimator = KerasRegressor(build_fn=create_lstm, verbose=1)
The following code is for my param grid and RandomSearch
lstm_param_grid = {
'dropout_rate': [0, 0.2, 0.4],
'neurons': [32, 64, 128],
'batch_size': [100, 200, 400],
'epochs': [50, 100, 150],
'lr': [1e-3, 1e-4, 1e-5]
}
lstm_RandomGrid = RandomizedSearchCV(estimator = lstm_estimator,
param_distributions = lstm_param_grid,
n_iter = 10,
verbose = 10,
n_jobs = -1,
cv = 5
)
In my create_lstm function the input shape of my data is equal to the window size and the number of features. After the RandomSearch i pass the arguments epochs and batch size when I fit the model. However with stateful=True you have to define batch_input_shape=(x, y, z).
I'm really not sure what exactly to do now. How can I change my code so that the RandomSearch still tests multiple batch sizes? And what exactly are (x,y,z) in my example? I tried (batch_size=100, window size, num of features) but that didn't work out.
Related
I am trying to tune the hyperparameters of MLP sequential model but getting an error while performing this task. I have tried degrading/upgrading the scikit-learn version and using np.asarray(X).astype(np.int) and np.asarray(X).astype(np.float32) but still getting the error. Please someone help me with how to fix this error. Thanks.
Error after using np.asarray(X).astype(np.int/float32)
---------------------------------------------------------------------------
TypeError Traceback (most recent call last)
<ipython-input-184-8cee47d11b3d> in <module>
1 x_norm_train=np.asarray(x_norm_train).astype(np.float32)
2
----> 3 y_train=np.asarray(y_train).astype(np.float32)
TypeError: float() argument must be a string or a number, not 'Timestamp'
Below is the code:
def mlp_tune():
def create_model(layers, activation, optimizer):
model = Sequential()
for i, nodes in enumerate(layers):
if i==0:
model.add(Dense(nodes, input_dim = x_norm_train.shape[1]))
model.add(Activation(activation))
else:
model.add(Dense(nodes))
model.add(Activation(activation))
model.add(Dense(1, activation = 'linear')) # Note: no activation beyond this point
model.compile(optimizer = optimizer, loss='mse')
# optimizers.Adam(learning_rate = rate, beta_1 = 0.9, \
# beta_2 = 0.999, amsgrad=False)
return model
model = KerasRegressor(build_fn = create_model, verbose=1)
# specifying layer architecture
optimizer = ['adam', 'rmsprop', 'sgd','adagrad', 'adadelta']
layers = [(3,), (10,), (30,), (10, 10), (10, 20), (20, 20), \
(30, 30), (10, 10, 10), (20, 20, 20), \
(30, 30, 30), (10, 20, 30), (20, 20, 30)]
activations = ['relu', 'tanh', 'sigmoid']
param_grid = dict(layers=layers, optimizer = optimizer, activation=activations, \
batch_size = [10, 50, 100], epochs=[10, 50])
grid = GridSearchCV(estimator = model, param_grid = param_grid,\
scoring='neg_mean_squared_error')
grid_result = grid.fit(x_norm_train, y_train)
[grid_result.best_score_, grid_result.best_params_]
testPredict = grid.predict(x_norm_test)
# model evaluation
print()
print(mean_squared_error(y_test, testPredict))
print()
# list all the data in history
print(history.history.keys())
# summarize history for accuracy
plt.figure(figsize=(12, 8))
plt.plot(grid_result.history['mean_squared_error'])
plt.plot(grid_result.history['val_mean_squared_error'])
plt.title('MLP Model Accuracy (After Hyperparameter tuning)', fontsize=18, y=1.03)
plt.ylabel('Accuracy')
plt.xlabel('Epoch')
plt.legend(['train', 'test'], loc='best')
plt.savefig("4 mlp model accuracy after tuning.png", dpi=300)
plt.show()
# summarize history for loss
plt.figure(figsize = (12, 8))
plt.plot(grid_result.history['loss'])
plt.plot(grid_result.history['val_loss'])
plt.title('MLP Model Loss (After Hyperparameter tuning)', fontsize=18, y=1.03)
plt.ylabel('Loss')
plt.xlabel('Epoch')
plt.legend(['train', 'test'], loc='best')
plt.savefig("5 mlp model loss after tuning.png", dpi=300)
plt.show()
# prepare data for plotting
y = y_test[:]
y.reset_index(inplace=True)
y.drop(['index'], axis = 1, inplace=True)
# plotting the results
sns.set_context('notebook', font_scale= 1.5)
plt.figure(figsize=(20, 10))
plt.plot(y['surge'])
plt.plot(testPredict, color= 'red')
plt.legend(['Observed Surge', 'Predicted Surge'],fontsize = 14)
plt.ylabel('Surge Height (m)')
plt.title("Observed vs. Predicted Storm Surge Height", fontsize=20, y=1.03)
plt.savefig("6 mlp observed vs predicted surge height (after tuning).png", dpi=300)
plt.show()
Error
ValueError: Failed to convert a NumPy array to a Tensor (Unsupported object type int).
The error may be faulty data pre processing;make sure that everything is properly formatted.
Below shows what the model expects as inputs:
[print(i.shape, i.dtype) for i in model.inputs]
[print(o.shape, o.dtype) for o in model.outputs]
[print(l.name, l.input_shape, l.dtype) for l in model.layers]
Pass the data to the model as the model expects. Thank You.
I'm using an ImageDataGenerator to input batches of images to a neural network, but can't work out the correct way to feed it. Running the following:
train_datagen = ImageDataGenerator(rescale=1./255, shear_range=0.2, zoom_range=0.2, horizontal_flip=True)
test_datagen = ImageDataGenerator(rescale=1./255)
training_set = train_datagen.flow_from_directory('/home/Training', target_size=(256,256), batch_size=32, class_mode='binary', color_mode = 'grayscale')
test_set = test_datagen.flow_from_directory('/home/Test', target_size=(256,256), batch_size=32, class_mode='binary',color_mode = 'grayscale' )
input_size = (256, 256, 1)
inputs = Input(input_size)
conv1 = Conv2D(64, 3, activation = 'relu', padding = 'same', kernel_initializer = 'he_normal')(inputs)
conv2 = Conv2D(2, 3, activation = 'relu', padding = 'same', kernel_initializer = 'he_normal')(conv1)
conv3 = Conv2D(1, 1, activation = 'sigmoid')(conv2)
model1 = Model(inputs = inputs, outputs = conv3)
model1.compile(optimizer = Adam(lr = 1e-4), loss = 'binary_crossentropy', metrics = ['accuracy'])
model1.fit_generator(training_set, steps_per_epoch=160, epochs=10, validation_data=test_set, validation_steps=800)
Results in:
Error when checking target: expected conv2d_198 to have 4 dimensions,
but got array with shape (14, 1)
It seems to use the batches as the input tensor, since removing all layers but the input layer results in a similar error. How can I correctly input them into the network?
Basically Keras is expecting your to pass your input dimensions and rows. Looks like you are passing an array with two dimensions. Can you make sure you are passing something like (-1, dimension 1, dimension 2, channels) ? you may need to use reshape. The -1 should tell Keras to infer the rows/observations. Im pretty new to Keras so I am sure someone else will have a better answer but you might be able to just do.. myinputarray.reshape()
Please forgive my ignorance as I am really new to the area. I am trying to get the correct output shape from my neural network which has 3 Conv2D layers then 2 Dense layers. My input shape is (140, 140, 4), which are 4 grayscale images. When I fit in 1 input, I am expecting an output of (1, 4) but I am getting a shape of (14, 14, 4) here. What am I doing wrong here? Thank you very much for your help in advance!
meta_layers = [Conv2D, Conv2D, Conv2D, Dense, Dense]
meta_inits = ['lecun_uniform'] * 5
meta_nodes = [32, 64, 64, 512, 4]
meta_filter = [(8,8), (4,4), (3,3), None, None]
meta_strides = [(4,4), (2,2), (1,1), None, None]
meta_activations = ['relu'] * 5
meta_loss = "mean_squared_error"
meta_optimizer=RMSprop(lr=0.00025, rho=0.9, epsilon=1e-06)
meta_n_samples = 1000
meta_epsilon = 1.0;
meta = Sequential()
meta.add(self.meta_layers[0](self.meta_nodes[0], init=self.meta_inits[0], input_shape=(140, 140, 4), kernel_size=self.meta_filters[0], strides=self.meta_strides[0]))
meta.add(Activation(self.meta_activations[0]))
for layer, init, node, activation, kernel, stride in list(zip(self.meta_layers, self.meta_inits, self.meta_nodes, self.meta_activations, self.meta_filters, self.meta_strides))[1:]:
if(layer == Conv2D):
meta.add(layer(node, init = init, kernel_size = kernel, strides = stride))
meta.add(Activation(activation))
elif(layer == Dense):
meta.add(layer(node, init=init))
meta.add(Activation(activation))
print("meta node: " + str(node))
meta.compile(loss=self.meta_loss, optimizer=self.meta_optimizer)
Your problem lies in the fact that in Keras with version >= 2.0, a Dense layer is applied to the last channel of the inputs (you may read about it here). So if you apply:
Dense(512)
to a Conv2D layer with shape (14, 14, 64) you'll get the output with shape (14, 14, 512) and then Dense(4) applied to it will give you output with shape (14, 14, 4). You can call model.summary() method to confirm my words.
In order to solve this you need to apply one of the following layers: GlobalMaxPooling2D, GlobalAveragePooling2D or Flatten to the output from the last convolutional layer in order to squash your output to be only 2 dimensional (with shape (batch_size, features).
I am training a CNN with Keras but with 30x30 patches from an image. I want to test the network with a full image but I get the following error:
ValueError: GpuElemwise. Input dimension mis-match. Input 2 (indices
start at 0) has shape[1] == 30, but the output's size on that axis is
100. Apply node that caused the error: GpuElemwise{Composite{((i0 + i1) - i2)}}[(0, 0)](GpuDimShuffle{0,2,3,1}.0, GpuReshape{4}.0,
GpuFromHost.0) Toposort index: 79 Inputs types:
[CudaNdarrayType(float32, 4D), CudaNdarrayType(float32, (True, True,
True, False)), CudaNdarrayType(float32, 4D)] Inputs shapes: [(10, 100,
100, 3), (1, 1, 1, 3), (10, 30, 30, 3)] Inputs strides: [(30000, 100,
1, 10000), (0, 0, 0, 1), (2700, 90, 3, 1)] Inputs values: ['not
shown', CudaNdarray([[[[ 0.01060364 0.00988821 0.00741314]]]]), 'not
shown'] Outputs clients:
[[GpuCAReduce{pre=sqr,red=add}{0,1,1,1}(GpuElemwise{Composite{((i0 +
i1) - i2)}}[(0, 0)].0)]]
This is my model.predict:
predict_image = model.predict(np.array([test_images[1]]), batch_size=1)[0]
It's seems like the issue is that the input size cannot be anything other than 30x30 but the first input shape for the first layer of my network is none, none, 3.
model.add(Convolution2D(n1, f1, f1, border_mode='same', input_shape=(None, None, 3), activation='relu'))
Is it simply not possible to test an image with different dimensions to the ones I trained with?
As fchollet himself described here, you should be able to define the input as so:
input_shape=(1, None, None)
However this will fail if you have layers that use the Flatten operation.
This suggests that you should be able to accomplish your goal with a fully convolutional NN.
I am new to the machine learning and TensorFlow. I am trying to train a simple model to recognize gender. I use small data-set of height, weight, and shoe size. However, I have encountered a problem with evaluating model's accuracy.
Here's the entire code:
import tflearn
import tensorflow as tf
import numpy as np
# [height, weight, shoe_size]
X = [[181, 80, 44], [177, 70, 43], [160, 60, 38], [154, 54, 37], [166, 65, 40],
[190, 90, 47], [175, 64, 39], [177, 70, 40], [159, 55, 37], [171, 75, 42],
[181, 85, 43], [170, 52, 39]]
# 0 - for female, 1 - for male
Y = [1, 1, 0, 0, 1, 1, 0, 0, 0, 1, 1, 0]
data = np.column_stack((X, Y))
np.random.shuffle(data)
# Split into train and test set
X_train, Y_train = data[:8, :3], data[:8, 3:]
X_test, Y_test = data[8:, :3], data[8:, 3:]
# Build neural network
net = tflearn.input_data(shape=[None, 3])
net = tflearn.fully_connected(net, 32)
net = tflearn.fully_connected(net, 32)
net = tflearn.fully_connected(net, 1, activation='linear')
net = tflearn.regression(net, loss='mean_square')
# fix for tflearn with TensorFlow 12:
col = tf.get_collection(tf.GraphKeys.TRAINABLE_VARIABLES)
for x in col:
tf.add_to_collection(tf.GraphKeys.VARIABLES, x)
# Define model
model = tflearn.DNN(net)
# Start training (apply gradient descent algorithm)
model.fit(X_train, Y_train, n_epoch=100, show_metric=True)
score = model.evaluate(X_test, Y_test)
print('Training test score', score)
test_male = [176, 78, 42]
test_female = [170, 52, 38]
print('Test male: ', model.predict([test_male])[0])
print('Test female:', model.predict([test_female])[0])
Even though model's prediction is not very accurate
Test male: [0.7158362865447998]
Test female: [0.4076206684112549]
The model.evaluate(X_test, Y_test) always returns 1.0. How do I calculate real accuracy on the test data-set using TFLearn?
You want to do binary classification in this case. Your network is set to perform linear regression.
First, transform the labels (gender) to categorical features:
from tflearn.data_utils import to_categorical
Y_train = to_categorical(Y_train, nb_classes=2)
Y_test = to_categorical(Y_test, nb_classes=2)
The output layer of your network needs two output units for the two classes you want to predict. Also the activation needs to be softmax for classification. The tf.learn default loss is cross-entropy and the default metric is accuracy, so this is already correct.
# Build neural network
net = tflearn.input_data(shape=[None, 3])
net = tflearn.fully_connected(net, 32)
net = tflearn.fully_connected(net, 32)
net = tflearn.fully_connected(net, 2, activation='softmax')
net = tflearn.regression(net)
The output will now be a vector with the probability for each gender. For example:
[0.991, 0.009] #female
Bear in mind that you will hopelessly overfit the network with your tiny data set. This means that during training the accuracy will approach 1 while, the accuracy on your test set will be quite poor.