I am training a Multi-layer Perceptron . I have two questions first one is that How can K fold prevents Overfitting because train-test-split also do same thing that take the training part and validate the model , same as for K fold instead of just there are multiple folds . But there is a chance of overfitting in train_test_split , then how K fold prevents it , because in my perception model could also gets overfit into train part of K fold what you think ?
Second Question is that i am getting 95% + accuracy from K fold , i have been told by sir that there is too much variance , how it is possible here because k fold resolves this overfitting?
K-Fold cross-validation won't reduce overfitting on its own, but using it will generally give you a better insight on your model, which eventually can help you avoid or reduce overfitting.
Using a simple training/validation split, the model may perform well if the way the split isn't indicative of the true data distribution. K-Fold cross-validation splits the data into k chunks & performs training k times, by using a particular chunk as the validation set & the rest of the chunks as the training set. Therefore, the model may perform quite well on some training fold, but relatively worse on other training folds. This will give you a better indication of how well the model truly performs.
If a relatively high training accuracy is attained but a substantially lower validation accuracy indicates overfitting (high variance & low bias). The goal would be to keep both variance & bias at low levels, potentially at the expense of slightly worse training accuracy, as this would indicate that the learnt model has generalised well to unseen instances. You can read more on the bias vs variance tradeoff.
Choosing the number of folds may also play a part in this insight, as explained in this answer. Depending on the size of the data, the training folds being used may be too large compared to the validation data.
K fold can help with overfitting because you essentially split your data into various different train test splits compared to doing it once. By running the train test splits on multiple different sets as opposed to just one, you get a better understanding of how your model is actually performing on the dataset and unseen data. It doesn’t completely prevent it and it all boils down to your data at the end of the day (if the data you have training, testing and validating is not truly representative of future points you can still end up with an overfit model).
Related
As I learned about cross-validation algorithm, from most of the articles on the web, there are variety of cross-validation methods. Here I want to be clear about the k-fold cross-validation technique.
In the k-fold cross-validation algorithm, we can split the training set in to k not-overlapped folds.
As we split the training data in to k folds, we have to train the model in k iterations.
So, in each iteration, we train the model with (k-1) folds and validate it with the remained fold.
In each split we can calculate the desired metric(s) of our model.
At the end we can report the training error by taking the average of scores of all iterations.
But what is the final trained model?
Some points in those articles are not clear for me?
Should I initiate model's parameters in each iteration?
I ask this, because if I don’t initialize the parameter's it could save the pattern of data which I want to be unseen in the next iteration and so on…
Should I save the initial parameter of the split in which I gained the best score, as the best initial values of the parameters?
Should I retrain the model initiating it with the initial values of the parameters gained in my second question and then feed it with whole training dataset and gain the final trained model?
Alright so before answering your question I will go a bit back to explain the purpose of cross validation and model evaluation. You can read these slides or research more about statistical learning theory if you want to go deeper.
Train/test split
Suppose you have a model with defined hyperparameter (or none) and you train it on the training split. If you calculate the metrics over the test split, this will give you the risk of the model on new data. Then you know that this particular model will perform like that on unseen data.
So we have a learning process B, that takes a dataset S (here the training dataset) as well as hyperparameters h, and gives a fitted model m; then B(S, h)->m (training B on S with hp h gives a model m, with its parameters). Then we tested this model to evaluate the risk R on the test dataset.
k-fold Cross validation
When doing k-fold cross validation, you fit k models using the learning process B. Each model is fitted on a different training set, and the risk is computed on non overlapping samples.
Then, you calculate the mean risk among the folds. A common mistake is that it gives you the performance of the model, that's not true. This gives you the mean (or expected) performances of the learning process B (and hyperparams h). That means, if you train a new model using B (and hyperparams h), its expected performance will be around the calculated metrics (of course this is not always true).
For your questions
Yes you should train the model from scratch, if possible with the same initial parameters (if initialization is not random) to avoid any difference between folds. Using a warm start with the previous parameters can modify the learning process, and the fitting.
No, if initialization is random let it be, if it is fixed use the same initial parameters for all folds
For the two previous questions, if by initial parameters you meant hyperparameters, then you should keep the same for all folds, otherwise the calculated risk will be useless. If you want to try multiple hyperparameters, you have to repeat the cross validation multiple times, and then you can select the best ones based on the risk calculated.
Once you tuned your hyperparameters you can train the model on your whole training set. This will give you a model m. Before your cross validation you can keep a small test split to evaluate this final model on unseen data
Let's say I initially split my dataset into training (80%) and test (20%) sets, perform a 10-fold CV on my training set and obtain an average R² of 75%. After that I check the best model's accuracy on the test set and obtain an R² of 74%, which indicates that the model is fairly robust. Now, before deploying it to real applications, I tune it with the whole data. Someone asks me the model's approximate R²; if I say 74% or 75%, I will me ignoring the fact that the model was now tunned with more data (test set). Is it a resonable approach to perform a leave one out CV on the chosen model with the whole data, compare the predicted targets with the real ones, check the R² (let's say it's 80% now) and say that the real-world model will most likely have an R² of 80%? I see no problems with that, but I do not know if this approach is correct.
It is true that you should train again on the whole data and it might lead to performance improvements. However, in this context your whole data should not be train + test! It should be just the trainign dataset but without any cross-validation. So before you had %80 for training and you were doing 10 fold CV, meaning that you were training your model actually on %72 of your complete data(train+test) and keeping the %8 for the validation. Now you should train it on the whole %80 percent and report your final results again on the unseen test set.
If you do LOOCV on the train + test, you can not report your performance on validation samples because this is how the model is finetuned and you might as well overfit to validation data.
For PyTorch's tutorial on performing transfer learning for computer vision (https://pytorch.org/tutorials/beginner/transfer_learning_tutorial.html), we can see that there is a higher validation accuracy than training accuracy. Applying the same steps to my own dataset, I see similar results. Why is this the case? Does it have something to do with ResNet 18's architecture?
Assuming there aren't bugs in your code and the train and validation data are in the same domain, then there are a couple reasons why this may occur.
Training loss/acc is computed as the average across an entire training epoch. The network begins the epoch with one set of weights and ends the epoch with a different (hopefully better!) set of weights. During validation you're evaluating everything using only the most recent weights. This means that the comparison between validation and train accuracy is misleading since training accuracy/loss was computed with samples from potentially much worse states of your model. This is usually most noticeable at the start of training or right after the learning rate is adjusted since the network often starts the epoch in a much worse state than it ends. It's also often noticeable when the training data is relatively small (as is the case in your example).
Another difference is the data augmentations used during training that aren't used during validation. During training you randomly crop and flip the training images. While these random augmentations are useful for increasing the ability of your network to generalize they aren't performed during validation because they would diminish performance.
If you were really motivated and didn't mind spending the extra computational power you could get a more meaningful comparison by running the training data back through your network at the end of each epoch using the same data transforms used for validation.
The short answer is that train and validation data are from different distributions, and it's "easier" for model to predict target in validation data then it is for training.
The likely reason for this particular case, as indicated by this answer, is data augmentation during training. This is a way to regularize your model by increasing variability in the training data.
Other architectures can use Dropout (or its modifications), which are deliberately "hurting" training performance, reducing the potential of overfitting.
Notice, that you're using pretrained model, which already contains some information about how to solve classification problem. If your domain is not that different from the data it was trained on, you can expect good performance off-the-shelf.
Does my model overfit? I would be sure it overfitted, if the validation loss increased heavily, while the training loss decreased. However the validation loss is nearly stable, so I am not sure. Can you please help?
I assume that you're using different hyperparameters? Perhaps save
the parameters and resume with a different set of hyperparameters.
This comment really depends on how you're doing hyperparameter
optimization.
Try with different training/test splits. It might be idiosyncratic.
Especially with so few epochs.
Depending on how costly it is to train the model and evaluate it,
consider bagging your models, akin to how a random forest operates.
In others words, fit your model to many different train/test splits,
and average the model outputs, either in terms of a majority
classification vote, or an averaging of the predicted probabilities.
In this case, I'd err on the side of a slightly overfit model,
because of the way that averaging can mitigate overfitting. But I
wouldn't train to death either, unless you're going to fit very very
many neural nets, and somehow ensure that you're decorrelating them
akin to the method of random subspaces from random forests.
Would you please guide me how to interpret the following results?
1) loss < validation_loss
2) loss > validation_loss
It seems that the training loss always should be less than validation loss. But, both of these cases happen when training a model.
Really a fundamental question in machine learning.
If validation loss >> training loss you can call it overfitting.
If validation loss > training loss you can call it some overfitting.
If validation loss < training loss you can call it some underfitting.
If validation loss << training loss you can call it underfitting.
Your aim is to make the validation loss as low as possible.
Some overfitting is nearly always a good thing. All that matters in the end is: is the validation loss as low as you can get it.
This often occurs when the training loss is quite a bit lower.
Also check how to prevent overfitting.
In machine learning and deep learning there are basically three cases
1) Underfitting
This is the only case where loss > validation_loss, but only slightly, if loss is far higher than validation_loss, please post your code and data so that we can have a look at
2) Overfitting
loss << validation_loss
This means that your model is fitting very nicely the training data but not at all the validation data, in other words it's not generalizing correctly to unseen data
3) Perfect fitting
loss == validation_loss
If both values end up to be roughly the same and also if the values are converging (plot the loss over time) then chances are very high that you are doing it right
1) Your model performs better on the training data than on the unknown validation data. A bit of overfitting is normal, but higher amounts need to be regulated with techniques like dropout to ensure generalization.
2) Your model performs better on the validation data. This can happen when you use augmentation on the training data, making it harder to predict in comparison to the unmodified validation samples. It can also happen when your training loss is calculated as a moving average over 1 epoch, whereas the validation loss is calculated after the learning phase of the same epoch.
Aurélien Geron made a good Twitter thread about this phenomenon. Summary:
Regularization is typically only applied during training, not validation and testing. For example, if you're using dropout, the model has fewer features available to it during training.
Training loss is measured after each batch, while the validation loss is measured after each epoch, so on average the training loss is measured ½ an epoch earlier. This means that the validation loss has the benefit of extra gradient updates.
the val set can be easier than the training set. For example, data augmentations often distort or occlude parts of the image. This can also happen if you get unlucky during sampling (val set has too many easy classes, or too many easy examples), or if your val set is too small. Or, the train set leaked into the val set.
If your validation loss is less than your training loss, you have not correctly split the training data. This correctly indicates that the distribution of the training and validation sets is different. It should ideally be the same. MOROVER, Good Fit: In the ideal case, the training and validation losses both drop and stabilize at specified points, indicating an optimal fit, i.e. a model that does neither overfit or underfit.