Using Accord.NET I've created a NaiveBayes classifier. It will classify a pixel based on 6 or so sets of image processing results. My images are 5MP, so a training set of 50 images creates a very large set of training data.
6 int array per pixel * 5 million pixels * 50 images.
Instead of trying to store all that data in memory, is there a way to incrementally train the NaiveBayes classifier? Calling Learn() multiple times overwrites the old data each time rather than adding to it.
Right now is not possible to train a Naive Bayes model incrementally using Accord.NET.
However, since all that Naive Bayes is going to do is to try to fit some distributions to your data, and since your data has very few dimensions, maybe you could try to learn your model on a subsample of your data rather than all of it at once.
When you go loading images to build your training set, you can try to randomly discard x% of the pixels in each image. You can also plot the classifier accuracy for different values of x to find the best balance between memory and accuracy for your model (hint: for such a small model and this large amount of training data, I expect that it wont make that much of a difference even if you dropped 50% of your data).
Related
I'm working on a side project based on multi-label classification. We consider images of 64x64 pixels made up of 4 thumbnails of 32x32 pixels that was randomly added together. The thumbnails are taken from the Cifar10 database, ending up with 40k train images, and 20k test images.
The initial goal of multi class classification becomes multilabel classification. Here is an example of the dataset.
The problem is that I tried a lot of things and the pure accuracy of the model doesn't exceed 1%, whereas the loss decreases.
Here is what I tried:
balancing the dataset ( same proportion of images regarding the class inside the image ).
Data augmentation up to 200k images in the train
Transfer Learning with dozens of models with/without fine tuning, and changed the last layer.
changing the multilabel problem to a multi class problem, I ended up with 385 classes that contains all the combinations ( I think ) of the images.
convolution 2D with a stride of 32 and a kernel size of 32x32.
Visio Transformer.
Trying dozens of optimizers, with different learning rate using a learning scheduler.
I'm pretty sure that the delimitation between the thumbnails is a problem for the convolution kernel because of the decor relation of the thumbnails in their corners.
I'm out of ideas that was I'm asking this question.
I have a question about how feature scaling works after training a model.
Let's say a neural network model predicts the height of a tree by training on outside temperature.
The lowest outside temperature in my training data is 60F and the max is 100F. I scale the temperature between 0 and 1 and train the model. I save the model for future predictions. Two months later, I want to predict on some new data. But this time the min and max temperatures in my test data are -20F and 50F, respectively.
How does the trained model deal with this? The range I imposed the scaling on in the training set to generate my trained model does not match the test data range.
What would prevent me from hard-coding a range to scale to that I know the data will always be within, say from -50F to 130F? The problem I see here is if I have a model with many features. If I impose a different hard scale to each feature, using feature scaling is essentially pointless, is it not?
Different scales won't work. Your model trains for one scale, it learns one scale, if you change the scale, your model will still think it's the same scale and make very shifted predictions.
Training again will overwrite what was learned before.
So, yes, hardcode your scaling (preferentially directly on your data, not inside the model).
And for a quality result, train with all the data you can gather.
I am reading Fit generator and data augmentation in keras, but there are still something that I am not quite sure about image augmentation in keras.
(1) In datagen.flow(), we also set a batch_size. I know batch_size is needed if we do mini-batch training, so are these two batch_size values the same, i mean, if we indicate batch_size in flow() generator, are we assuming we will do mini-batch training with the same batch_size?
(2)
Let me assume the size of training set is 10,000. I guess the only difference between model.fit_generator() and model.fit() at each epoch is that, for the former one, we are using 10,000 of randomly transformed images, rather than the original 10,000 ones. But for other epochs, we are using another 10,000 images which are totally different than those used in the first epoch, because all the images are randomly generated. Is it right?
It is like we are always using new images at each epoch, which is different from the ordinary case, when the same set of images are used at each epoch.
I am new to this area. Please help!
the 1st question:the answer is YES.
the 2nd question:yes we are always using new images at each epoch,if we use data augmentation in model.fit_generator()
I trained a CNN (on tensorflow) for digit recognition using MNIST dataset.
Accuracy on test set was close to 98%.
I wanted to predict the digits using data which I created myself and the results were bad.
What I did to the images written by me?
I segmented out each digit and converted to grayscale and resized the image into 28x28 and fed to the model.
How come that I get such low accuracy on my data set where as such high accuracy on test set?
Are there other modifications that i'm supposed to make to the images?
EDIT:
Here is the link to the images and some examples:
Excluding bugs and obvious errors, my guess would be that your problem is that you are capturing your hand written digits in a way that is too different from your training set.
When capturing your data you should try to mimic as much as possible the process used to create the MNIST dataset:
From the oficial MNIST dataset website:
The original black and white (bilevel) images from NIST were size
normalized to fit in a 20x20 pixel box while preserving their aspect
ratio. The resulting images contain grey levels as a result of the
anti-aliasing technique used by the normalization algorithm. the
images were centered in a 28x28 image by computing the center of mass
of the pixels, and translating the image so as to position this point
at the center of the 28x28 field.
If your data has a different processing in the training and test phases then your model is not able to generalize from the train data to the test data.
So I have two advices for you:
Try to capture and process your digit images so that they look as similar as possible to the MNIST dataset;
Add some of your examples to your training data to allow your model to train on images similar to the ones you are classifying;
For those still have a hard time with the poor quality of CNN based models for MNIST:
https://github.com/christiansoe/mnist_draw_test
Normalization was the key.
I am trying to classify road signs. For this reason I want to train Hog descriptors with the use of SVMs. I have extracted the hog descriptors for training data with dimensions 64x64. The positive training data are 60% and the negative 40% of the whole sample.When I am traing using SVM of opencv (with a linear kernel) evereything seems fine, but when I am trying to predict, the results fail and show only one class (the result is always 1). I have tried to feed my data into SVMlight as well, and all the negatives are missclassified.Any ideas what could be possible wrong? Maybe the small number of training data? (I am just trying to implement the code and see that everything is fine without using a the training data).