I am doing image classification. Firstly I am feeding my images to my CNN model in Keras.
I want to add new features at the output of Flatten layer in keras and then feed it to the dense layer(s). How do I write a code for it?
Basically I am using Convolution for images and then at the end I want to add other features like Age Sex etc.
max_pool_final = MaxPooling2D(pool_size=(2,2))(conv_final)
flat = Flatten()(max_pool_final)
dense = Dense(128)(flat)
Before feeding flat as an input to the dense layer, I want to add a few features to flat. How do I do this?
Thanks for help!
You just have to use the Concatenate layer to append those features to the flattened vector with a new Input layer:
otherInp = Input(shape = (n_features, ))
concatenatedFeatures = Concatenate(axis = 1)([flat, otherInp])
dense = Dense(128)(concatenatedFeatures)
Related
I have two files with me, an input file and an output file. The input file goes through a transformation logic and produces the output file. The issue here is that, I am not aware of the transformation logic between the input and output file.
The input file contains 10 fields and output file contains 7 fields. These 10 fields are transformed into 7 fields using a transformation logic.
Is there way using some machine learning algorithm, to build a model that automatically deduce the relationship between input and output and will be able to predict the output based on the data in the input file?
I think I have something, that might help you solve your issue:
You have various inputs with different datatypes. Also you have different outputs with different datatypes. Lets have this dataset as example when working with tensorflow and keras:
x_categorical=[1,2,3,4,5]
x_categorical_2=np.random.choice(x_categorical, len(x_categorical))
x_continuus=np.random.random_sample(len(x_categorical))
y_categorical = [0,2,3,4,5]
y_continuus = np.random.random_sample(len(x_categorical))
Create the tf.data.Datasets and zip the x and y values together, so it fits the model input:
ds_x = tf.data.Dataset.from_tensor_slices(x_categorical)
ds_x1 = tf.data.Dataset.from_tensor_slices(x_categorical_2)
ds_x2 = tf.data.Dataset.from_tensor_slices(x_continuus)
dataset_x = tf.data.Dataset.zip((ds_x,ds_x1,ds_x2))
ds_y = tf.data.Dataset.from_tensor_slices(y_categorical)
ds_y1 = tf.data.Dataset.from_tensor_slices(y_continuus)
dataset_y = tf.data.Dataset.zip((ds_y,ds_y1))
dataset_train = tf.data.Dataset.zip((dataset_x, dataset_y))
Build an example model, which concatenates the inputs, has one layer which contains the "logic" for the data combination and two more layers, that have a logic for each output:
from tensorflow.keras import layers as layer
layer_input_categorical = layer.Input(shape=(1),name="x_categorical", dtype=tf.float32)
layer_input_categorical_2 = layer.Input(shape=(1),name="x_categorical_2", dtype=tf.float32)
layer_input_continuus = layer.Input(shape=(1),name="x_continuus", dtype=tf.float32)
concat_layer = layer.Concatenate()([layer_input_categorical,layer_input_categorical_2, layer_input_continuus])
dense_layer = layer.Dense(100)(concat_layer)
dense_layer_out_cat = layer.Dense(50)(dense_layer)
dense_layer_out_con = layer.Dense(50)(dense_layer)
output_categorical = layer.Dense(5, activation="softmax")(dense_layer_out_cat)
output_continuus = layer.Dense(1, activation="sigmoid")(dense_layer_out_con)
model = tf.keras.Model(inputs=[layer_input_categorical, layer_input_categorical_2, layer_input_continuus], \
outputs=[output_categorical, output_continuus])
model.compile(optimizer="Nadam", loss=["mse","sparse_categorical_crossentropy"])
Please note the usage of two loss functions (mse for regression and sparse_categorical_crossentropy for classification.
Also note, that the two output layers have different activation functions, softmax for classification (for each class, you get the probability) and one sigmoid for regression.
At the end, just start the training with:
model.fit(dataset_train.batch(1), epochs=20)
For sure this is not the best way of doing it, but it proves, that is is possible.
I am looking for a way to re initialize layer's weights in an existing keras pre trained model.
I am using python with keras and need to use transfer learning,
I use the following code to load the pre trained keras models
from keras.applications import vgg16, inception_v3, resnet50, mobilenet
vgg_model = vgg16.VGG16(weights='imagenet')
I read that when using a dataset that is very different than the original dataset it might be beneficial to create new layers over the lower level features that we have in the trained net.
I found how to allow fine tuning of parameters and now I am looking for a way to reset a selected layer for it to re train. I know I can create a new model and use layer n-1 as input and add layer n to it, but I am looking for a way to reset the parameters in an existing layer in an existing model.
For whatever reason you may want to re-initalize the weights of a single layer k, here is a general way to do it:
from keras.applications import vgg16
from keras import backend as K
vgg_model = vgg16.VGG16(weights='imagenet')
sess = K.get_session()
initial_weights = vgg_model.get_weights()
from keras.initializers import glorot_uniform # Or your initializer of choice
k = 30 # say for layer 30
new_weights = [glorot_uniform()(initial_weights[i].shape).eval(session=sess) if i==k else initial_weights[i] for i in range(len(initial_weights))]
vgg_model.set_weights(new_weights)
You can easily verify that initial_weights[k]==new_weights[k] returns an array of False, while initial_weights[i]==new_weights[i] for any other i returns an array of True.
Its commonplace for various neural network architectures in NLP and vision-language problems to tie the weights of an initial word embedding layer to that of an output softmax. Usually this produces a boost to sentence generation quality. (see example here)
In Keras its typical to embed word embedding layers using the Embedding class, however there seems to be no easy way to tie the weights of this layer to the output softmax. Would anyone happen to know how this could be implemented ?
Be aware that Press and Wolf dont't propose to freeze the weights to some pretrained ones, but tie them. That means, to ensure that input and output weights are always the same during training (in the sense of synchronized).
In a typical NLP model (e.g. language modelling/translation), you have an input dimension (vocabulary) of size V and a hidden representation size H. Then, you start with an Embedding layer, which is a matrix VxH. And the output layer is (probably) something like Dense(V, activation='softmax'), which is a matrix H2xV. When tying the weights, we want that those matrices are the same (therefore, H==H2).
For doing this in Keras, I think the way to go is via shared layers:
In your model, you need to instantiate a shared embedding layer (of dimension VxH), and apply it to either your input and output. But you need to transpose it, to have the desired output dimensions (HxV). So, we declare a TiedEmbeddingsTransposed layer, which transposes the embedding matrix from a given layer (and applies an activation function):
class TiedEmbeddingsTransposed(Layer):
"""Layer for tying embeddings in an output layer.
A regular embedding layer has the shape: V x H (V: size of the vocabulary. H: size of the projected space).
In this layer, we'll go: H x V.
With the same weights than the regular embedding.
In addition, it may have an activation.
# References
- [ Using the Output Embedding to Improve Language Models](https://arxiv.org/abs/1608.05859)
"""
def __init__(self, tied_to=None,
activation=None,
**kwargs):
super(TiedEmbeddingsTransposed, self).__init__(**kwargs)
self.tied_to = tied_to
self.activation = activations.get(activation)
def build(self, input_shape):
self.transposed_weights = K.transpose(self.tied_to.weights[0])
self.built = True
def compute_mask(self, inputs, mask=None):
return mask
def compute_output_shape(self, input_shape):
return input_shape[0], K.int_shape(self.tied_to.weights[0])[0]
def call(self, inputs, mask=None):
output = K.dot(inputs, self.transposed_weights)
if self.activation is not None:
output = self.activation(output)
return output
def get_config(self):
config = {'activation': activations.serialize(self.activation)
}
base_config = super(TiedEmbeddingsTransposed, self).get_config()
return dict(list(base_config.items()) + list(config.items()))
The usage of this layer is:
# Declare the shared embedding layer
shared_embedding_layer = Embedding(V, H)
# Obtain word embeddings
word_embedding = shared_embedding_layer(input)
# Do stuff with your model
# Compute output (e.g. a vocabulary-size probability vector) with the shared layer:
output = TimeDistributed(TiedEmbeddingsTransposed(tied_to=shared_embedding_layer, activation='softmax')(intermediate_rep)
I have tested this in NMT-Keras and it trains properly. But, as I try to load a trained model, it gets an error, related to the way Keras loads the models: it doesn't load the weights from the tied_to. I've found several questions regarding this (1, 2, 3), but I haven't managed to solve this issue. If someone have any ideas on the next steps to take, I'd be very glad to hear them :)
As you may read here you should simply set trainable flag to False. E.g.
aux_output = Embedding(..., trainable=False)(input)
....
output = Dense(nb_of_classes, .. ,activation='softmax', trainable=False)
I'm looking to do some image classification on PDF documents that I convert to images. I'm using tensorflow inception v3 pre trained model and trying to retrain the last layer with my own categories following the tensorflow tuto. I have ~1000 training images per category and only 4 categories. With 200k iterations I can reach up to 90% of successful classifications, which is not bad but still need some work:
The issue here is this pre-trained model takes only 300*300p images for input. Obviously it messes up a lot with the characters involved in the features I try to recognize in the documents.
Would it be possible to alter the model input layer so I can give him images with better resolution ?
Would I get better results with a home made and way simpler model ?
If so, where should I start to build a model for such image classification ?
If you want to use a different image resolution than the pre-trained model uses , you should use only the convolution blocks and have a set of fully connected blocks with respect to the new size. Using a higher level library like Keras will make it a lot easier. Below is an example on how to do that in Keras.
import keras
from keras.layers import Flatten,Dense,GlobalAveragePooling2D
from keras.models import Model
from keras.applications.inception_v3 import InceptionV3
base_model = InceptionV3(include_top=False,input_shape=(600,600,3),weights='imagenet')
x = base_model.output
x = GlobalAveragePooling2D()(x)
x = Dense(1024,activation='relu')(x)
#Add as many dense layers / Fully Connected layers required
pred = Dense(10,activation='softmax')(x)
model = Model(base_model.input,pred)
for l in model.layers[:-3]:
l.trainable=False
The input_top = False will give you only the convolution blocks. You can use the input_shape=(600,600,3) to set the required shape you want. And you can add a couple of dense blocks/Fully connected blocks/layers to the model.The last layer should contain the required number of categories .10 represent the number of classes.By this approach you use all the weights associated with the convolution layers of the pre trained model and train only the last dense layers.
I have an 2D input (or 3D if one consider the number of samples) and I want to apply a keras layer that would take this input and outputs another 2D matrix. So, for example, if I have an input with size (ExV), the learning weight matrix would be (SxE) and the output (SxV). Can I do this with Dense layer?
EDIT (Nassim request):
The first layer is doing nothing. It's just to give an input to Lambda layer:
from keras.models import Sequential
from keras.layers.core import Reshape,Lambda
from keras import backend as K
from keras.models import Model
input_sample = [
[[1,2,3,4,5],[6,7,8,9,10],[11,12,13,14,15],[16,17,18,19,20]]
,[[21,22,23,24,25],[26,27,28,29,30],[31,32,33,34,35],[36,37,38,39,40]]
,[[41,42,43,44,45],[46,47,48,49,50],[51,52,53,54,55],[56,57,58,59,60]]
]
model = Sequential()
model.add(Reshape((4,5), input_shape=(4,5)))
model.add(Lambda(lambda x: K.transpose(x)))
intermediate_layer_model = Model(input=model.input,output=model.layers[0].output)
print "First layer:"
print intermediate_layer_model.predict(input_sample)
print ""
print "Second layer:"
intermediate_layer_model = Model(input=model.input,output=model.layers[1].output)
print intermediate_layer_model.predict(input_sample)
It depends on what you want to do. Is it 2D because it's a sequence? Then LSTM are made for that and will return a sequence if desired size if you set return_sequence=True.
CNN's can also work on 2D inputs and will output something of variable size depending on the number of kernels you use.
Otherwise you can reshape it to a (E x V, ) 1D tensor, use a Dense layer with SxV dimension and reshape the output to a (S,V) 2D tensor...
I can not help you more, we need to know your usecase :-) there are too many possibilities with neural networks.
EDIT :
You can use TimeDistributed(Dense(S)).
If your input has a shape (E,V), you reshape to (V,E) to have V as the "time dimension". Then you apply TimeDistributed(Dense(S)) which will be a dense layer with weights (ExS), the output will have the shape (V,S) so you can reshape it to (S,V).
Does that make what you want ? The TimeDistributed() layer will apply the same Dense(S) layer to every V lines of your input with shared weights.
EDIT 2:
After looking at the code of keras backend, it turns out that to use the transpose from tensorflow with 'permutation patterns' option available, you need to use K.permute_dimensions(x,pattern). The batch dimension must be included. In your case :
Lambda(lambda x: K.permute_dimensions(x,[0,2,1]))
K.transpose(x) uses the same function internally (for tf backend) but permutations is set to the default value which is [n,n-1,...,0].
What you want is probably autoencoder.
https://blog.keras.io/building-autoencoders-in-keras.html