I trained my custom data set in the yoloV5s model and I got 80% accuracy on my inference. Now I need to increase the accuracy by adding more images and labels.
My question here is, I already trained 10,000+ labels to reach 80% it took 7 hours for me. Shall I need to include the old 10,000+ data with my new data which is only 1000 to train and improve my accuracy?
Is there any way that I can include the new data only to retrain the model even I add a new class?
How can I save my time and space?
The question you are asking is of topic continual learning, which is an active area of research nowadays. Since you need to add more classes to your model, you need to add the new class with the previous data and retrain the model from start. If you don't do that, i.e., you only train on the new class, your model will forget completely about the previous data (learned feature); this forgetting is known as Catastrophic Forgetting.
Many people have suggested various ways to avoid this Catastrophic forgetting; I personally feel that Progressive Neural Network is highly immune to Forgetting. Apart from it, you can find other methods here
As I told you, this is currently a highly active area of research; There is no full-proof solution. For now, the best way is to add the new data to the previous data and retrain your model.
Related
I used yolov5 to train an object detection model. is it possible to add more annotated images after i have already trained the original model or must i restart the whole training with the new set of images?
You are asking about continual learning - this is a very active field of research, and there is no single solution/method to tackle it. You'll have to do more research to find the right approach for your specific settings.
Closed. This question needs to be more focused. It is not currently accepting answers.
Want to improve this question? Update the question so it focuses on one problem only by editing this post.
Closed 2 years ago.
Improve this question
I am working on modeling a dataset from object detection. I am relatively new to deep learning. I am having a hard time extending the idea of cross-validation in the context of deep learning. Usually, the train time is huge with deep network and k-fold CV is not a reasonable approach. So, probably 1-fold cross-validation makes more sense (I have seen people use this in practice). I am trying to reason this choice and thinking about the idea behind cross-validation: hyper-parameter tuning, or quantify when the modeling starts to over-fit. My questions are the following:
What about the random sampling error with a 1-fold CV? My thoughts: with k-fold CV this error is averaged out when k>1. Also, with k=1, the hyper-parameter also doesn't seem reasonable to me: the values we end up finding can be coupled with the (random) sample we called validation set. So, what's the point of a 1-fold CV?
There's already a crunch of data points in the data I am working with. I have around ~4k images, 2 categories (object+background), bounding boxes for each image. I think it's common wisdom that deep networks learn better with more data. Why would I want to reduce my training set by keeping aside a validation set in this context? I don't see any clear advantages. On the contrary, it seems like using the entire dataset to train can lead to a better object detection model. If this is true, then how would one know when to stop, i.e. I could keep training, without any feedback into whether the model has started overfitting?
How are production models deployed? I guess I have never thought about this one much while taking courses. The approach was pretty clear that you always have a train, validation, test set. In actual settings, how do you leverage the entire data to create a production model? (probably connected to #2, i.e. dealing with practical aspects like how much to train etc.)
Public Computer Vision datasets in the domain of Object Detection are usually large enough that this isn't an issue. How much of an issue it is in your scenario can be shown by the gap in performance between validation and test set. Cross validation with n = 1 essentially means having a fixed validation set.
You want to keep the validation set in order to tune the parameters of your model. Increasing the number of weights will surely increase the performance on the training set but you want to check how this behaves on unseen data e.g. the validation set. That said, many people will tune parameters according to the performance on the validation set and then do one more training where they combine the training and validation data before finally testing it on the test set.
I think this is already answered in 2. You can extend this by training on all 3 sets but whatever performance you achieve on it will not be representative. The number of epochs/iterations you want to train for should therefore be decided before merging the data.
You have to decide what it is you want to optimize for. Most papers optimize for performance on the test set which is why it should never be used for training or validating parameter choices. In reality you might often favour a "better" model by including validation and test data into the training. You will never know how much "better" this model is until you find another test set. You're also risking that something "strange" will happen when including the test data. You're essentially training with closed eyes.
I am looking to implement machine learning for a problems that are built on small data sets related to approvals of expenses in a specific supply chain domain. Typically labelled data is unavailable
I was looking to build models in one data set that I have labelled data and then use that model developed in similar contexts- where the feature set is very similar, but not identical. The expectation is that this allows the starting point for recommendations and gather labelled data in the new context.
I understand this is the essence of Transfer Learning. Most of the examples I read in this domain speak of image data sets- any guidance how this can be leveraged in small data sets using standard tree-based classification algorithms
I can’t really speak to tree-based algos, I don’t know how to do transfer learning with them. But, for deep learning models, the customary method for transfer learning is to load up a pretrained model, then retrain the last layer of the dataset using your new data, and then fine-tune the rest of the network.
If you don’t have much data to go on, you might look into creating synthetic data.
raghu, I believe you are looking for a kernel method when you are saying abstraction layer in deep learning. There are several ML algorithms that support kernel functions. With kernel functions, you might be able to do it; but using kernel functions might be more complex than solving your original problem. I would lean toward Tdoggo's suggestion of using Decision Tree.
Sorry, I want to add a comment, but they won't allow me, so I posted a new answer.
Ok with tree-based algos you can do just what you said: train the tree on one dataset and apply it to another similar dataset. All you would need to do is change the terms/nodes on the second tree.
For instance, let’s say you have a decision tree trained for filtering expenses for a construction company. You will outright deny any reimbursements for workboots, because workers should provide those themselves.
You want to use the trained tree on your accounting firm, and so instead of workboots, you change that term to laptops, because accountants should be buying their own.
Does that make sense, and is that helpful to you?
After some research, we have decided to proceed with random forest models with the intuition that trees in the original model that have common features will form the starting point for decisions.
As we gain more labelled data in the new context, we will start replacing the original trees with new trees that comprise of (a)only new features and (b) combination of old and new features
This has worked to provide reasonable results in initial trials
I have a model, that needs to train with real world data that I am acquiring daily. In every 3 or 4 days, I can prepare around 500 images for training. So, I must start training and checking the model just after getting 500 images. Meanwhile I will acquire another 500 images and so on. Whether it is OK to train with first 500 data set and save the model weights and continue train with latest 500 data set by using saved weights?
This is basically like transfer learning. You take a pre-trained model and fine-tune it on your new data. You will have to save the model and its weight and then load them back and train on the new data like you would normally. This is a common practice.
You have two options - effectively engage in transfer learning (as mentioned above) OR, if you really believe old data + new data = the best possible data set for you to train on, consider retraining from scratch on the complete data set (old data + new data). The latter gives all data, new and old, an equally fair shake, which is not necessarily true of transfer learning. Although I have to question your need to do this every 3 or 4 days - if your problem is well formulated and your model design is good, at some point you should have enough data that the model trained on that data generalizes well enough that continuously giving it more data will no longer improve the performance significantly. Also, if the model will perform significantly better having been trained on 2000 images than 500 images, why not just wait a couple more weeks until you have 2000 images before releasing it into the real world? Obviously this depends on your task and area of industry, so you may well have a good reason that I'm not aware of, but it's worth thinking about.
Can you have TOO MUCH training data or not?
I am working on a system that will update training data when a user gives it feedback of a mistake it has made in an attempt to not make the same mistake again (i.e if the user looks a little different to their usual training images, it will add the new capture of them to training data).
Will this decrease performance at all? Should there be a maximum? Would it be better just to have the same training set and just accept the fail rate instead of trying to improve it?
Cheers!
Depending on how different the user looks, this could be a problem.
lets say the user is wearing sunglasses, looks the wrong way,and wears a scarf.
This would occlude too much of the image to properly determine if this is a face or not.
Training on such images would provide horrendous results overall, because they are not something that qualifies as a face, or at least not according to the theories provided for eigenfaces.
If you want to keep training a model according to feedback, I think you should at least have a person check the images and decide if they are worth training.
But, if you have trained the model with a proper dataset to begin with, almost all the feedback you would receive would never properly qualify as a face. because if they did, the model would not have failed in the first place.
regarding a maximum, If I recall correctly, there is not a hard limit you should respect, but up to a certain point, the amount of time needed to retrain the model would become absurtly long, which could be unwanted for your specific situation.
I hope this made any sense to you, If you have any more questions about my answer, just leave a comment.