I am new to the community and I am trying to learn as much as possible but I came to a point where I need help because I don't manage to find the right solution.
I have a CNN with a softmax layer at the end but I would like to have an SVM and I really don't know how to do it. I found something online saying that basically, I have to change from softmax to linear but I don't understand why and since I don't understand I can not do it.
If anyone knows how to do it please can you let me know how to do it with the code below? Thank you and sorry if this question was asked before or if you find it not that smart. I am new to this.
Code below:
model = keras.models.Sequential([
keras.layers.Conv2D(filters=16, kernel_size=(3,3), activation='relu', input_shape=(IMG_HEIGHT,IMG_WIDTH,channels)),
keras.layers.Conv2D(filters=32, kernel_size=(3,3), activation='relu'),
keras.layers.MaxPool2D(pool_size=(2, 2)),
keras.layers.BatchNormalization(axis=-1),
keras.layers.Conv2D(filters=64, kernel_size=(3,3), activation='relu'),
keras.layers.Conv2D(filters=128, kernel_size=(3,3), activation='relu'),
keras.layers.MaxPool2D(pool_size=(2, 2)),
keras.layers.BatchNormalization(axis=-1),
keras.layers.Flatten(),
keras.layers.Dense(512, activation='relu'),
keras.layers.BatchNormalization(),
keras.layers.Dropout(rate=0.5),
keras.layers.Dense(43, activation='softmax')
])
You don't need any layers after a flatten layer.Now use this model to extract a feature vector for xtrain and xtest for svm .
Refer: YouTube channel digitalsreeni
Related
I am starting to make a neural network that can learn chess. As of current, my training data is roughly 50 million lines long and stored in a CSV file, where each line contains a fen and an outcome. I've made a model and a small function so far. It can play, but not very well.
def create_model() -> tf.keras.Model:
"""Create and return a TensorFlow model for evaluating chess positions.
Returns:
A TensorFlow model.
"""
model = tf.keras.Sequential()
model.add(tf.keras.layers.Conv2D(filters=128, kernel_size=3, padding='same', activation='relu', kernel_regularizer='l2', input_shape=(8, 8, 12)))
model.add(tf.keras.layers.BatchNormalization())
model.add(tf.keras.layers.Conv2D(filters=128, kernel_size=3, padding='same', activation='relu', kernel_regularizer='l2'))
model.add(tf.keras.layers.BatchNormalization())
model.add(tf.keras.layers.MaxPool2D(pool_size=2))
model.add(tf.keras.layers.Dropout(rate=0.2))
model.add(tf.keras.layers.Conv2D(filters=256, kernel_size=3, padding='same', activation='relu', kernel_regularizer='l2'))
model.add(tf.keras.layers.BatchNormalization())
model.add(tf.keras.layers.Conv2D(filters=256, kernel_size=3, padding='same', activation='relu', kernel_regularizer='l2'))
model.add(tf.keras.layers.BatchNormalization())
model.add(tf.keras.layers.MaxPool2D(pool_size=2))
model.add(tf.keras.layers.Dropout(rate=0.2))
model.add(tf.keras.layers.Conv2D(filters=512, kernel_size=3, padding='same', activation='relu', kernel_regularizer='l2'))
model.add(tf.keras.layers.BatchNormalization())
model.add(tf.keras.layers.Conv2D(filters=512, kernel_size=3, padding='same', activation='relu', kernel_regularizer='l2'))
model.add(tf.keras.layers.BatchNormalization())
model.add(tf.keras.layers.MaxPool2D(pool_size=2))
model.add(tf.keras.layers.Dropout(rate=0.2))
model.add(tf.keras.layers.Flatten())
model.add(tf.keras.layers.Dense(units=1024, activation='relu', kernel_regularizer='l2'))
model.add(tf.keras.layers.BatchNormalization())
model.add(tf.keras.layers.Dropout(rate=0.2))
model.add(tf.keras.layers.Dense(units=512, activation='relu', kernel_regularizer='l2'))
model.add(tf.keras.layers.BatchNormalization())
model.add(tf.keras.layers.Dropout(rate=0.2))
model.add(tf.keras.layers.Dense(units=2, activation='softmax', kernel_regularizer='l2'))
optimiser = tf.keras.optimizers.Adam()
model.compile(optimizer=optimiser, loss='categorical_crossentropy', metrics=['accuracy'])
return model
def train() -> None:
"""
Train the TensorFlow model using the data in the `sample_fen.csv` file. The model is saved to the file `weights.h5` after training.
"""
# This is only needed for training
import pandas as pd
training_data = pd.read_csv(r'neural_net\Players\mtcs_engine\sample_fen.csv', chunksize=100000)
model = create_model()
try:
model.load_weights(r"neural_net\Players\mtcs_engine\weights.h5")
print("Weights file found. Loading weights.")
except FileNotFoundError:
print("No weights file found. Training from scratch.")
try:
for cycle, chunk in enumerate(training_data):
games = chunk.values.tolist()
if cycle <= 11:
continue
# Preprocess the data
positions = []
outcomes = []
for game in games:
position = fen_to_tensor(game[0])
outcome = game[1]
if outcome == "w":
one_hot_outcome = [1, 0]
elif outcome == "b":
one_hot_outcome = [0, 1]
else:
one_hot_outcome = [0, 0]
outcomes.append(one_hot_outcome)
positions.append(position)
positions = np.array(positions)
outcomes = np.array(outcomes)
model.fit(positions, outcomes, epochs=150, batch_size=64)
print(f"Finished training cycle {cycle}")
except KeyboardInterrupt:
pass
model.save_weights(r"neural_net\Players\mtcs_engine\weights.h5")
print()
print("Saved weights to disk")
but upon learning for around a day its accuracy has increased from 0.5000 to 0.5100 with a loss in the hundreds of thousands. To be honest, I'm not really sure what I'm doing at all. Does anyone have any pointers, be it with the model or anything else? Full code can be found at https://github.com/Iridum-png/warden-chess/blob/master/neural_net/Players/mtcs_engine/mtcs_engine.py
I think you should think more about what it is you're trying to accomplish. Working in AI is difficult but far from impossible for beginners, but it's important to remember to start small. That is to say, answer these questions in order:
What is the model's goal?
How can it best achieve it?
What do I need to do to handle the model's inputs and outputs to realistically make these predictions?
It looks like all your model is trying to do is determine if a game is a win for white or for black. Is that what you want it to do?
You also need to do quite a bit of research into what each layer in your network is for. In particular: this stretch is highly problematic and will lead to a very confused output due to your use of dropouts (which are good for preventing overfitting but it seems the opposite is the case for you):
model.add(tf.keras.layers.BatchNormalization())
model.add(tf.keras.layers.Dropout(rate=0.2))
model.add(tf.keras.layers.Dense(units=512, activation='relu', kernel_regularizer='l2'))
model.add(tf.keras.layers.BatchNormalization())
model.add(tf.keras.layers.Dropout(rate=0.2))
If you want this model to PLAY chess I would suggest having your network do the following:
Take as input the 8,8 chess board, but also provide it with a move. This way the model can output a probability of the move instead of trying to guess at what to play (which is much more difficult). You can also provide the color to play for additional aid
Research the monte-carlo search and how alpha-zero implemented it
All in all, making a move prediction from only a board, while possible, is extremely difficult, and would require 2000+ epochs to have a hope of a chance at, and even then, the model would be woefully inept at looking ahead and wouldn't get much above 600 ELO if I had to guess.
I am trying to have two sequential models with one input and one output, I am not sure if this is possible as I've been researching Keras etc. and all that I seem to be investigating is how to do two and three inputs with multiple outputs which is what I DO NOT WANT.
To explain, from the 1st sequential model level output AT THE FLATTEN LAYER, I am trying to convert the activations back to a cube shape to be passed as input to the 2nd level sequential model making this a 2 level multi-level structure. I have my diagram to assist with the understanding of my thoughts, please review such in the link.
My apologies in advance for the lack of understanding and explanations, I am not sure of the terminology for such a model or approach hence my diagram to clarify my thoughts at this stage. All I came up with is a multi-level sequential model and its fusuion mechanisms. I searched for examples of such but thus far my efforts were unsuccessful.
THANK YOU'ssssss in advance for any assistance given.
enter image description here
I was following this code but it leads to a the fusion of two inputs. I would like to use the last flattened layer of the 1st level as input for the 2nd level, then process onwards towards classification. I am not sure if this is possible please assist?
model = Sequential()
model.add(Conv3D(32, kernel_size=(3, 3, 3),activation='relu',
input_shape=input_shape))
model.add(Conv3D(64, (3, 3, 3), activation='relu'))
model.add(MaxPooling3D(pool_size=(2, 2, 2)))
model.add(Dropout(0.25))
model.add(Flatten())
print(model.output_shape)
# The additional data (the coordinates x,y,z)
extra = Sequential()
extra.add(Activation('sigmoid', input_shape=(3,)))
print(extra.output_shape)
merged = Concatenate([model, extra])
# New model should encompass the outputs of the convolutional
network and the coordinates that have been merged.
# But how?
new_model = Sequential()
new_model.add(Dense(128, activation='relu'))
new_model.add(Dropout(0.8))
new_model.add(Dense(32, activation='sigmoid'))
new_model.add(Dense(num_classes, activation='softmax'))
new_model.compile(loss=keras.losses.categorical_crossentropy,
optimizer=keras.optimizers.Adadelta(),
metrics=['accuracy'])
First of all i'm very new to the field. maybe my question is a bit too naive of even trivial..
I'm currently trying to understand how can i go about recognizing different faces.
Here is what i tried so far and the main issues with each approach:
1) Haar Cascade -> HOG -> SVM:
The main issue is that the algorithm becomes very indecisive when more than 4 people are trained.. the same occurs when we change Haar Cascade for a pre-trained CNN to detect faces..
2) dlib facial landmarks -> distance between points -> SVM or Simple Neural Network Classification:
This is the current approach and it behaves very well when when 4 people are trained.. when more people are trained it becomes very messy, jumping from decision to decision and never resolves to a choice.
I've read online that Triplet loss is the way to go.. but I very confused as to how id go about implementing it.. can i use the current distance vectors found using Dlib or should i scrap everything and train my own CNN?
If i can use the distance vectors how would i pass the data to the algorithm? is Triplet loss a trivial neural network only with it's loss function altered?
I've took the liberty to show exactly how the distance vectors are being calculated:
The green lines represent the distances being calculated
A 33 float list is returned which is then fed to the classifier
Here is the relevant code for the classifier (Keras):
def fit_classifier(self):
x_train, y_train = self._get_data(self.train_data_path)
x_test, y_test = self._get_data(self.test_data_path)
encoding_train_y = np_utils.to_categorical(y_train)
encoding_test_y = np_utils.to_categorical(y_test)
model = Sequential()
model.add(Dense(10, input_dim=33, activation='relu'))
model.add(Dense(20, activation='relu'))
model.add(Dense(30, activation='relu'))
model.add(Dense(40, activation='relu'))
model.add(Dense(30, activation='relu'))
model.add(Dense(20, activation='relu'))
model.add(Dense(10, activation='relu'))
model.add(Dense(max(y_train)+1, activation='softmax'))
model.compile(loss='mse', optimizer='adam', metrics=['accuracy'])
model.fit(x_train, encoding_train_y, epochs=100, batch_size=10)
I think this is a more theoretical question than anything else.. if someone with good experience in the field could help me out i'd be very happy!
I'm looking for implementations of convolutional autoencoder using MxNet. But there is only one example of autoencoder based on Fully Connected Networks, which is here. There is also an issue asking similar questions in github, but receives very few responses. Is there any toy example of convolutional autoencoders implemented using MxNet?
Please find an example of Conv Autoencoder model in Mxnet Gluon. Code quoted from here. Training this model in a standard way in Gluon.
from mxnet import gluon as g
class CNNAutoencoder(g.nn.HybridBlock):
def __init__(self):
super(CNNAutoencoder, self).__init__()
with self.name_scope():
self.encoder = g.nn.HybridSequential('encoder_')
with self.encoder.name_scope():
self.encoder.add(g.nn.Conv2D(16, 3, strides=3, padding=1, activation='relu'))
self.encoder.add(g.nn.MaxPool2D(2, 2))
self.encoder.add(g.nn.Conv2D(8, 3, strides=2, padding=1, activation='relu'))
self.encoder.add(g.nn.MaxPool2D(2, 1))
self.decoder = g.nn.HybridSequential('decoder_')
with self.decoder.name_scope():
self.decoder.add(g.nn.Conv2DTranspose(16, 3, strides=2, activation='relu'))
self.decoder.add(g.nn.Conv2DTranspose(8, 5, strides=3, padding=1, activation='relu'))
self.decoder.add(g.nn.Conv2DTranspose(1, 2, strides=2, padding=1, activation='tanh'))
def forward(self, x):
x = self.encoder(x)
x = self.decoder(x)
return x
model = CNNAutoencoder()
model.hybridize()
There is still no convolutional autoencoder example in mxnet, though there is some progress in research in that area. Anyway, there is a ticket for that in MxNet github, but it is still open. You are more than welcome to contribute, by, for example, migrating the code from Keras.
I have a question about using Keras to which I'm rather new. I'm using a convolutional neural net that feeds its results into a standard perceptron layer, which generates my output. This CNN is fed with a series of images. This is so far quite normal.
Now I like to pass a short non-image input vector directly into the last perceptron layer without sending it through all the CNN layers. How can this be done in Keras?
My code looks like this:
# last CNN layer before perceptron layer
model.add(Convolution2D(200, 2, 2, border_mode='same'))
model.add(Activation('relu'))
model.add(MaxPooling2D(pool_size=(2, 2), strides=(2, 2)))
model.add(Dropout(0.25))
# perceptron layer
model.add(Flatten())
# here I like to add to the input from the CNN an additional vector directly
model.add(Dense(1500, W_regularizer=l2(1e-3)))
model.add(Activation('relu'))
model.add(Dropout(0.5))
model.add(Dense(1))
Any answers are greatly appreciated, thanks!
You didn't show which kind of model you use to me, but I assume that you initialized your model as Sequential. In a Sequential model you can only stack one layer after another - so adding a "short-cut" connection is not possible.
For this reason authors of Keras added option of building "graph" models. In this case you can build a graph (DAG) of your computations. It's a more complicated than designing a stack of layers, but still quite easy.
Check the documentation site to look for more details.
Provided your Keras's backend is Theano, you can do the following:
import theano
import numpy as np
d = Dense(1500, W_regularizer=l2(1e-3), activation='relu') # I've joined activation and dense layers, based on assumption you might be interested in post-activation values
model.add(d)
model.add(Dropout(0.5))
model.add(Dense(1))
c = theano.function([d.get_input(train=False)], d.get_output(train=False))
layer_input_data = np.random.random((1,20000)).astype('float32') # refer to d.input_shape to get proper dimensions of layer's input, in my case it was (None, 20000)
o = c(layer_input_data)
The answer here works. It is more high level and works also for the tensorflow backend:
input_1 = Input(input_shape)
input_2 = Input(input_shape)
merge = merge([input_1, input_2], mode="concat") # could also to "sum", "dot", etc.
hidden = Dense(hidden_dims)(merge)
classify = Dense(output_dims, activation="softmax")(hidden)
model = Model(input=[input_1, input_2], output=hidden)