How to give variable size images as input in keras - image-processing

I am writing a code for image classification for two classes using keras with tensorflow backend. My images are stored in folder in computer and i want to give these images as input to my keras model. load_img takes only one input image so i have to use either flow(x,y) or flow_from_directory(directory), but in flow(x,y) we need to also provide labels which is length task so i am using flow_from_directory(directory). My images are of variable sizes like 20*40, 55*43..... but here it is mentioned that fixed target_size is required. In this solution it is given that we can give variable size images as input to convolution layer using input_shape=(1, None, None) or input_shape=(None,None,3) (channel last and color images) but fchollet mention that it is not useful for flatten layer and my model consist both convolution and flatten layers. In that post only moi90 suggest that try different batches but every batch should have images with same size, but it is not possible me to group images with same sizes because my data is very scatter. So i decided to go with batch size=1 and write following code:
from __future__ import print_function
from keras.models import Sequential
from keras.layers import Dense, Dropout, Flatten, Conv2D, MaxPooling2D
from keras import backend as K
import numpy as np
from keras.preprocessing.image import ImageDataGenerator
input_shape = (None,None,3)
model = Sequential()
model.add(Conv2D(8, kernel_size=(3, 3),
activation='relu',
input_shape=input_shape))
model.get_weights()
model.add(Conv2D(16, (3, 3), activation='relu'))
model.add(MaxPooling2D(pool_size=(2, 2)))
model.add(Dropout(0.25))
model.add(Flatten())
model.add(Dense(32, activation='relu'))
model.add(Dropout(0.5))
model.add(Dense(2, activation='softmax'))
model.compile(loss='binary_crossentropy',optimizer='rmsprop',metrics=['accuracy'])
train_datagen = ImageDataGenerator()
test_datagen = ImageDataGenerator()
train_generator = train_datagen.flow_from_directory('/data/train', target_size=input_shape, batch_size=1,class_mode='binary')
validation_generator = test_datagen.flow_from_directory('/data/test',target_size=input_shape,batch_size=1,class_mode='binary')
model.fit_generator(train_generator,steps_per_epoch=1,epochs=2,validation_data=validation_generator,validation_steps=1)
Now i am getting following error:
Traceback (most recent call last):
File "<ipython-input-8-4e22d22e4bd7>", line 23, in <module>
model.add(Flatten())
File "/home/nd/anaconda3/lib/python3.6/site-packages/keras/models.py", line 489, in add
output_tensor = layer(self.outputs[0])
File "/home/nd/anaconda3/lib/python3.6/site-packages/keras/engine/topology.py", line 622, in __call__
output_shape = self.compute_output_shape(input_shape)
File "/home/nd/anaconda3/lib/python3.6/site-packages/keras/layers/core.py", line 478, in compute_output_shape
'(got ' + str(input_shape[1:]) + '. '
ValueError: The shape of the input to "Flatten" is not fully defined (got (None, None, 16). Make sure to pass a complete "input_shape" or "batch_input_shape" argument to the first layer in your model.
I am sure it is not because of img_dim_ordering and backend but because of this i have checked both are th Please help to correct he code or help how i can give variable size images as input to my model.

You can train variable sizes, as long as you don't try to put variable sizes in a numpy array.
But some layers do not support variable sizes, and Flatten is one of them. It's impossible to train models containing Flatten layers with variable sizes.
You can try, though, to replace the Flatten layer with either a GlobalMaxPooling2D or a GlobalAveragePooling2D layer. But these layers may condense too much information into a small data, so it might be necessary to add more convolutions with more channels before them.
You must make sure that your generator will produce batches containing images of the same size, though. The generator will fail when trying to put two or more images with different sizes in the same numpy array.

See the answer in https://github.com/keras-team/keras/issues/1920
Yo you should change the input to be:
input = Input(shape=(None, None,3))
The in the end add GlobalAveragePooling2D():
Try something like that ...
input = Input(shape=(None, None,3))
model = Sequential()
model.add(Conv2D(8, kernel_size=(3, 3),
activation='relu',
input_shape=(None, None,3))) #Look on the shape
model.add(Conv2D(16, (3, 3), activation='relu'))
model.add(MaxPooling2D(pool_size=(2, 2)))
model.add(Dropout(0.25))
# IMPORTANT !
model add(GlobalAveragePooling2D())
# IMPORTANT !
model.add(Flatten())
model.add(Dense(32, activation='relu'))
model.add(Dropout(0.5))
model.add(Dense(2, activation='softmax'))
model.compile(loss='binary_crossentropy',optimizer='rmsprop',metrics=['accuracy'])

Unfortunately you can't train a neural network with various size images as it is. You have to resize all images to a given size. Fortunately you don't have to do this in your hard drive, permanently by keras does this for you on hte fly.
Inside your flow_from_directory you should define a target_size like this:
train_generator = train_datagen.flow_from_directory(
'data/train',
target_size=(150, 150), #every image will be resized to (150,150) before fed to neural network
batch_size=32,
class_mode='binary')
Also, if you do so, you can have whatever batch size you want.

Related

Validation accuracy and validation loss almost remains constant in every epoch

I am making an autonomous farming robot for my final year project. I want to move it autonomously in lanes in side the farms. I am just using the raspberry pi image in front of my vehicle. I collect my data through pi and then send it to my computer for training.
Initially i have just trained it for moving in a straight line. As i have not used encoders in my motors so there is a possibility of its being diverging along one direction , so i have to constantly give it the feedback to stay on the right path.
Sample image is as follows, Note this is black and white image :enter image description here
I have 836 images for training and 356 for validation. When i am trying to train it, my model accuracy doesnot improves much. I have tried changing different structures, from fully connected layers to different convolutional layers, my training accuracy doesnot improves much and perhaps most of the times validation accuracy and validation loss remains same.
I am confused that why is this so, is this to do with my code or should i apply computer vision techniques on the image so that features are more prominently visible. What should be the best approach to tackle this problem.
My code is as follows:
import numpy
from keras.models import Sequential
from keras.layers import Dense
from keras.layers import Dropout
from keras.layers import Flatten
from keras.layers.convolutional import Conv2D
from keras.layers.convolutional import MaxPooling2D
# fix dimension ordering issue
from keras import backend as K
import numpy as np
import glob
import pandas as pd
from sklearn.model_selection import train_test_split
K.set_image_dim_ordering('th')
# fix random seed for reproducibility
seed = 7
numpy.random.seed(seed)
def load_data(path):
print("Loading training data...")
training_data = glob.glob(path)[0]
data=np.load(training_data)
a=data['train']
b=data['train_labels']
s=np.concatenate((a, b), axis=1)
data=pd.DataFrame(s)
data=data.sample(frac=1)
X = data.iloc[:,:-4]
y=data.iloc[:,-4:]
print("Image array shape: ", X.shape)
print("Label array shape: ", y.shape)
# normalize data
# train validation split, 7:3
return train_test_split(X, y, test_size=0.3)
data_path = "*.npz"
X_train,X_test,y_train,y_test=load_data(data_path)
# reshape to be [samples][channels][width][height]
X_train = X_train.values.reshape(X_train.shape[0], 1, 120, 320).astype('float32')
X_test = X_test.values.reshape(X_test.shape[0], 1, 120, 320).astype('float32')
# normalize inputs from 0-255 to 0-1
X_train = X_train / 255.0
X_test = X_test / 255.0
# one hot encode outputs
num_classes = y_test.shape[1]
# define a simple CNN model
def baseline_model():
model = Sequential()
model.add(Conv2D(30, (5, 5), input_shape=(1, 120, 320), activation='relu'))
model.add(MaxPooling2D(pool_size=(2, 2)))
model.add(Conv2D(15, (3, 3), activation='relu'))
model.add(MaxPooling2D(pool_size=(2, 2)))
model.add(Dropout(0.2))
model.add(Flatten())
model.add(Dense(128, activation='relu'))
model.add(Dense(50, activation='relu'))
model.add(Dense(num_classes, activation='softmax'))
# Compile model
model.compile(loss='categorical_crossentropy', optimizer='adam', metrics=['accuracy'])
return model
# build the model
model = baseline_model()
# Fit the model
model.fit(X_train, y_train, validation_data=(X_test, y_test), epochs=10, batch_size=10)
# Final evaluation of the model
scores = model.evaluate(X_test, y_test, verbose=0)
print("CNN Error: %.2f%%" % (100-scores[1]*100))
sample output: This is the best output and it is of the above code:
enter image description here
I solved this problem by changing the structure of my algorithm and using NVIDIA's deep learning car algorithm to solve this problem. The algorithm is very robust and applies basic computer vision also on it. You can easily find sample implementation for toy cars on medium/youtube also.
this article was really helpful for me:
https://towardsdatascience.com/deeppicar-part-1-102e03c83f2c
additionally this resource was also very helpful:
https://zhengludwig.wordpress.com/projects/self-driving-rc-car/

Layer Counting with Keras Deep Learning

I am working on my First deep-learning project on counting layers in an image with convolutional neural network.
After fixing tons of errors, I could finally train my model. However, I am getting 0 accuracy; after 2nd epoch it just stops because it is not learning anything.
Input will be a 1200 x 100 size image of layers and output will be an integer.
If anyone can look over my model and can suggest a tip. That will be awesome.
Thanks.
from keras.layers import Reshape, Conv2D, MaxPooling2D, Flatten
model = Sequential()
model.add(Convolution2D(32, 5, 5, activation='relu', input_shape=(1,1200,100)))
model.add(MaxPooling2D(pool_size=(2, 2)))
model.add(Convolution2D(64, 5, 5, activation='relu'))
model.add(MaxPooling2D(pool_size=(2, 2)))
model.add(Flatten())
model.add(Dense(1, activation='relu'))
batch_size = 1
epochs = 10
sgd = SGD(lr=0.01, decay=1e-6, momentum=0.9, nesterov=True)
model.compile(sgd, loss='poisson', metrics=['accuracy'])
earlyStopping=keras.callbacks.EarlyStopping(monitor='val_loss', patience=0, verbose=0, mode='auto')
history = model.fit(xtrain, ytrain, batch_size=batch_size, nb_epoch=epochs, validation_data=validation, callbacks=[earlyStopping], verbose=1)
There are sooo many thing to criticise?
1200*100 size of an image (I assume that they're pixels) is so big for CNN's. In ImageNet competitions, images are all 224*224, 299*299.
2.Why don't you use linear or sigmoid activation on last layer?
Did you normalize your outputs between 0 and 1? Normalize it, just divide your output with the maximum of your output and multiply with the same number when using your CNN after training/predicting.
Don't use it with small data, unnecessary :
earlyStopping=keras.callbacks.EarlyStopping(monitor='val_loss', patience=0, verbose=0, mode='auto')
Lower your optimizer to 0.001 with Adam.
Your data isn't actually big, it should work, probably your problem is at normalization of your output/inputs, check for them.

Wrong predictions with own MNIST-like image data in trained CNN model

I have made a CNN model to predict number, trained by MNIST data. Using keras wrapper for tensorflow. I am having trouble in predicting my own input data. I have trained the model with epochs =100 and tested model with test set of MNIST which is working fine with accuracy of 97% approx. I have saved this model as 'my_model_conv2d.h5'.
1st Code:
# importing modules
import numpy as np
import keras
import matplotlib.pyplot as plt
from keras.optimizers import SGD
from keras.models import Sequential
from keras.layers import Dense,Activation, Dropout, Flatten
from keras.layers import Conv2D, MaxPooling2D
from keras.datasets import mnist
from keras.utils import np_utils
plt.rcParams['figure.figsize'] = (7,7)
#reading MNIST data
(X_train,y_train),(X_test,y_test)=mnist.load_data('mnist.npz')
print X_train.shape, " train datad shape"
print y_train.shape, " labels shape"
#copying data for plotting
test=X_test.copy()
#reshaping data type according to the tensorflow param.
X_train=X_train.reshape(X_train.shape[0],X_train.shape[1],X_train.shape[2],1)
X_test=X_test.reshape(X_test.shape[0],X_test.shape[1],X_test.shape[2],1)
input_shape=(X_train.shape[1],X_train.shape[2],1)
X_train=X_train.astype('float32')
X_test=X_test.astype('float32')
X_train/=255
X_test/=255
# one_hot vector
Y_train=np_utils.to_categorical(y_train,10)
Y_test=np_utils.to_categorical(y_test,10)
print X_train.shape, " train datad shape"
print y_train.shape, " labels shape"
#building CNN model
model = Sequential()
model.add(Conv2D(32, (3, 3), input_shape=input_shape))
model.add(Activation('relu'))
model.add(Conv2D(32, (3, 3)))
model.add(Activation('relu'))
model.add(MaxPooling2D(pool_size=(2, 2)))
model.add(Dropout(0.25))
model.add(Flatten())
model.add(Dense(128))
model.add(Activation('relu'))
model.add(Dropout(0.5))
model.add(Dense(10))
model.add(Activation('softmax'))
#optimizer type initialization
sgd=SGD(lr=0.01, decay=1e-6, momentum=0.9, nesterov=True)
#compiling model
model.compile(loss='categorical_crossentropy', optimizer='sgd')
#training model
model.fit(X_train,Y_train, batch_size=128,epochs=100, verbose=1,validation_data=(X_test,Y_test))
#evaluating
score=model.evaluate(X_test,Y_test,verbose=1)
print score, "score"
#predicting classes using MNIST test data
predicted_classes=model.predict_classes(X_test)
print predicted_classes.shape, "predicted_classes.shape"
correct_indices = np.nonzero(predicted_classes == y_test)[0]
incorrect_indices = np.nonzero(predicted_classes != y_test)[0]
#saving model for future use
model.save('my_model_conv2d.h5')
print len(correct_indices), "no . of correct samples"
print len(incorrect_indices), "no . of incorrect samples"
print str((len(incorrect_indices)/float(len(y_test)))*100)+'%', "error percentage"
#plotting graph of predicted test data
plt.figure()
for i, correct in enumerate(correct_indices[:9]):
plt.subplot(3,3,i+1)
plt.imshow(test[correct], cmap='gray', interpolation='none')
plt.title("Predicted {}, Class {}".format(predicted_classes[correct], y_test[correct]))
plt.show()
plt.figure()
for i, incorrect in enumerate(incorrect_indices[:9]):
plt.subplot(3,3,i+1)
plt.imshow(test[incorrect], cmap='gray', interpolation='none')
plt.title("Predicted {}, Class {}".format(predicted_classes[incorrect], y_test[incorrect]))
plt.show()
I have made second python code which opens the model 'my_model_conv2d.h5'. In this code, I have also made an interactive window to draw a number, which is used as an input image for the prediction. I have taken care of background and font color of image and also the size (28,28), approximately it's similar to MNIST data.
2nd Code:
import cv2
import numpy as np
from keras.models import Sequential,load_model
drawing=False
mode=True
#function for mouse events
def interactive_drawing(event,x,y,flags,param):
global ix,iy,drawing, mode
if event==cv2.EVENT_LBUTTONDOWN:
drawing=True
ix,iy=x,y
print "EVENT_LBUTTONDOWN"
elif event==cv2.EVENT_MOUSEMOVE:
if drawing==True:
print "EVENT_MOUSEMOVE"
if mode==True:
print "EVENT_MOUSEMOVE"
cv2.line(img,(ix,iy),(x,y),1,1)
ix=x
iy=y
elif event==cv2.EVENT_LBUTTONUP:
drawing=False
print "EVENT_LBUTTONUP"
if mode==True:
print "EVENT_LBUTTONUP"
cv2.line(img,(ix,iy),(x,y),1,1)
ix=x
iy=y
return x,y
#function for predicting number
def cnn(img):
im=img.copy()
#loading cnn model
model = load_model('my_model_conv2d.h5')
img=img.reshape(1,28,28,1).astype('float32')
#prediction of class using drawn image as an input
predicted_class=model.predict_classes(img)
print predicted_class, "class"
cv2.imwrite('actual=9:predicted='+str(predicted_class)+'.jpg',im*255)
#image same as mnist image
img = np.zeros((28,28), 'float32')
cv2.namedWindow('Drawing_window')
cv2.setMouseCallback('Drawing_window',interactive_drawing)
while(1):
cv2.imshow('Drawing_window',img*255)
k=cv2.waitKey(1)&0xFF
if k==27:
break
cv2.destroyAllWindows()
#calling cnn function for prediction
cnn(img)
Most of the predictions are wrong with my input data. Also, I have made fully connected layer before implementing CNN, but the result was same. So, I tried CNN but the problem is same. I have checked the test data of MNIST in the second code which is working fine. You can check the result here https://drive.google.com/open?id=1E26zinOLrMw7XKhsHd9vnSYoHXzadiJ3 . Please check the image name, name of the image will suggest actual and predicted value.
Where I am doing wrong please suggest.
You need to use the exact same preprocessing applied to the images during training to images during inference.
This is why your program normalizes the train and test data in the same way.
X_train/=255
X_test/=255
This is also true to any other preprocessing you might want to do such as PCA or normalization by z-score.
So make sure that, in your case, your input image is in the range (0,1) by dividing it by 255 (if a range of 0 to 255 is to be expected from that opencv call)
Edit:
I just went ahead and trained the model and tried your program. Indeed it seems to make alot of mistakes (much more that what's to be expected since it got 1% validation error).
My guess is that since mnist is already somewhat preprocessed,
The digits have been size-normalized and centered in a fixed-size
image.
your model might expect your test images to be preprocessed as well.
I would recommend then using some mnist variant with more noise/rotation that might mitigate the problem. Eg: mnist-rot,mnist-back-rand.

How to set initial weights in Convolution2D in Keras?

The syntax for adding Convolution2D layer is Keras is
https://keras.io/layers/convolutional/#convolution2d. I am unable to pass "weights" argument correctly. How should I do that?
conv1_1 = Conv2D(64, kernel_size=(3, 3), activation='relu', padding='same',
weight_and_bias=[weight, biases], name='conv1_1')(input)
The shape of weight is (nb_filter, nb_channel, filter_size, filter_size), shape of biases is (nb_channel,)
You should pass a list of numpy arrays to set as initial weights.
For Convolution2D, weights list have two items, one in shape (nb_filter, nb_channel, nb_row, nb_col) and bias in shape (nb_filter,).
According to the author of Keras:
If you have doubts about what these shapes are, you can simply
instantiate your layer then call get_weights(), and look at the
output. The argument weights, and also the method
set_weights(weights), expect exactly the same format as the output
of get_weights().

Keras: How to feed input directly into other hidden layers of the neural net than the first?

I have a question about using Keras to which I'm rather new. I'm using a convolutional neural net that feeds its results into a standard perceptron layer, which generates my output. This CNN is fed with a series of images. This is so far quite normal.
Now I like to pass a short non-image input vector directly into the last perceptron layer without sending it through all the CNN layers. How can this be done in Keras?
My code looks like this:
# last CNN layer before perceptron layer
model.add(Convolution2D(200, 2, 2, border_mode='same'))
model.add(Activation('relu'))
model.add(MaxPooling2D(pool_size=(2, 2), strides=(2, 2)))
model.add(Dropout(0.25))
# perceptron layer
model.add(Flatten())
# here I like to add to the input from the CNN an additional vector directly
model.add(Dense(1500, W_regularizer=l2(1e-3)))
model.add(Activation('relu'))
model.add(Dropout(0.5))
model.add(Dense(1))
Any answers are greatly appreciated, thanks!
You didn't show which kind of model you use to me, but I assume that you initialized your model as Sequential. In a Sequential model you can only stack one layer after another - so adding a "short-cut" connection is not possible.
For this reason authors of Keras added option of building "graph" models. In this case you can build a graph (DAG) of your computations. It's a more complicated than designing a stack of layers, but still quite easy.
Check the documentation site to look for more details.
Provided your Keras's backend is Theano, you can do the following:
import theano
import numpy as np
d = Dense(1500, W_regularizer=l2(1e-3), activation='relu') # I've joined activation and dense layers, based on assumption you might be interested in post-activation values
model.add(d)
model.add(Dropout(0.5))
model.add(Dense(1))
c = theano.function([d.get_input(train=False)], d.get_output(train=False))
layer_input_data = np.random.random((1,20000)).astype('float32') # refer to d.input_shape to get proper dimensions of layer's input, in my case it was (None, 20000)
o = c(layer_input_data)
The answer here works. It is more high level and works also for the tensorflow backend:
input_1 = Input(input_shape)
input_2 = Input(input_shape)
merge = merge([input_1, input_2], mode="concat") # could also to "sum", "dot", etc.
hidden = Dense(hidden_dims)(merge)
classify = Dense(output_dims, activation="softmax")(hidden)
model = Model(input=[input_1, input_2], output=hidden)

Resources