I work with TFF 0.12.0, with the same architecture of ResNet50, I execute an example of IID dataset, and an example with non-IID dataset, but I find that accuracy of non-IID dataset is better (high) than accuracy of IID dataset especially in the first rounds.
Related
Why is the accuracy on train dataset not always 100% while we use the same dataset to train the model?
Though tree-based ML algorithms give us 100% accuracy on train dataset many times, but why is this not happening every time. I know this results in overfitting but why not 100% accuracy every time on the dataset using which our model is trained?
I have trained LSTM AND decision tree on my data set (type of text classification). I have used K-cross fold validation with k=10.
Decision tree accuracy 61%
LSTM accuracy 90%
Now when I predict on totally unseen data then decision tree predicts more well and good as compared to LSTM.
Why it happens? If LSTM accuracy is more then why decision tree performs more well on unseen data as compare to LSTM?
Your LSTM model may have greater accuracy than a decision tree when training, but the fact that it doesn't generalize well to unseen data, indicates that the LSTM is overfitting to the training data. Try adjusting the train-validation split and batch size to see if that improves your models.
The validation loss during training would indicate which model is better. You can also try using random forests (cluster of decision trees) which has been known to give better results than one decision tree alone
I'm working on a project with colorectal cancer stage multiclass-classification using Gene Expression Data. My dataset contains 11 Biomarkers. The results from the classification are around 40%. I have tried different models for classification with KNN, SVM, neural network..., and also I have tried algorithms from ensemble machine learning. Has anyone has any idea what can I do with the dataset to improve the results?
To decide what to do next, you will need some metrics:
How well can a team of human experts classify the data?
What is the model accuracy on the training dataset?
What is the model accuracy on the testing dataset?
If the training accuracy is much worse than human experts, you should increase the complexity of the model until the training results approach or exceed human experts. You can do this by increasing the number of input features, choosing a different machine learning model, or increasing the number of layers in the NN. If the training accuracy is poor, you need to improve this first before spending time improving the testing accuracy.
If the training accuracy is good but the testing accuracy is much worse than the training accuracy, you are probably overfitting. Get or create more training data, and use regularization.
I'm working on relation classification with the SemEval2010 Task 8 dataset. The dataset is already split into 8'000 samples for the training and 2'717 for the testing. In order to be as fair as possible, I use only my model at the end to computing its performance (F1-Score).
In order to tune my convolutional neural networks, I keep 6'400 samples for the training and 1'600 for the validation. I train the model and after each epoch (~10' of computation) I compute the F1-Score of my predictions.
I read the paper http://page.mi.fu-berlin.de/prechelt/Biblio/stop_tricks1997.pdf and stop training when the last 3 performances were increasing (similar to UP in the paper). In the paper, they return the model corresponding to the best performance seen so far.
My question is : in order to be as accurate as possible, we need the whole 8'000 samples for the training. Is it correct to say we will train until the epoch which had the best performance on the validation set and then do the predictions ? Or should we save the model corresponding to the best performance and "waste" 1'600 samples ?
I have been working on the Sentiment analysis prediction using the Rotten Tomatoes movie reviews dataset.
The dataset has 5 classes {0,1,2,3,4} where 0 being very negative and 4 being very positive
The dataset is highly unbalanced,
total samples = 156061
'0': 7072 (4.5%),
'1': 27273 (17.4%),
'2': 79583 (50.9%),
'3': 32927 (21%),
'4': 9206 (5.8%)
as you can see class 2 has almost 50% samples and 0 and 5 contribute to ~10% of training set
So there is a very strong bias for class 2 thus reducing the accuracy of classification for class 0 and 4.
What can I do to balance the dataset? One solution would be to get equal number of samples by reducing the samples to only 7072 for each class, but it reduces the dataset drastically!
How can I optimize and balance the dataset without affecting the accuracy of overall classification?
You should not balance the dataset, you should train a classifier in a balanced manner. Nearly all existing classifiers can be trained with some cost sensitive objective. For example - SVMs let you "weight" your samples, simply weight samples of the smaller class more. Similarly Naive Bayes has classes priors - change them! Random forest, Neural networks, Logistic regression, they all let you somehow "weight" samples, it is the core technique for getting more balanced results.
For classification problems, you can try class_weight='balanced' option in your estimator, such as Logistic, SVM, etc. For example:
http://scikit-learn.org/stable/modules/generated/sklearn.linear_model.LogisticRegression.html#sklearn.linear_model.LogisticRegression