If I want to use manual learning rate, for example, initial learning rate is 0.01, after 10000 steps, decay learning rate to 0.001, and after 20000 steps, decay learning rate to 0.0001.
How can I know these steps I need to decay learning rate? Is there any experience can guide me?
We usually can find these steps in paper, but how does author get these steps?
A good initial learning rate is one that is as high as possible without causing the error to diverge. This is found by trial and error. Guess some typical initial learning rate like 0.01, run a few epochs and see what happens. If the error is diverging then divide by 10, otherwise multiply by 10. Settle on the highest value before you run into divergent error.
A good time to lower the learning rate is when you notice the error does not seem to be improving anymore. For example you run 100 epochs at lr=0.01 and the error is plateauing. Try reducing your learning rate to 0.001.
You can also consider using learning rate decay where you set a schedule to decay the learning rate automatically.
Related
I train a ResNet50 model with TFF, I use test accuracy on test data for evaluation, but I find many fluctuations as shown in the figure below, So please how can I avoid this fluctuation ?
I would say behavior such as this is to be expected for stochastic optimization in general. The inherent variance causes you to oscillate somewhere around good solution. The magnitude of the variance and properties of the optimization objective control how much this oscillates when looking at a accuracy metric.
For plain SGD, decreasing learning rate decreases the variance and slows down convergence.
For optimization methods for federated learning, the story is a bit more complicated, but decreasing the client learning rate, or decreasing the number of local steps (while keeping other things the same) can have a similar effect, typically including slowing down convergence. More details can be found in https://arxiv.org/abs/2007.00878 mentioned also in the other answer. Potentially decreasing the client learning rate across rounds could also work. The details can differ also based on what exactly is the optimization method you are using.
How is the test accuracy calculated? How many local epochs are the clients training?
If the global model is tested on a held out set of examples, it is possible that clients are detrimentally overfitting during local training. As the global model approaches convergence, each client ends up training a model that works well for them individually, but may be diverging from the optimal global model (sometimes called client drift https://arxiv.org/abs/1910.06378). This is may occur when the client's local dataset has a distribution very different from the global distribution and more likely when the client learning rates are high (https://arxiv.org/abs/2007.00878).
Decreasing the client learning rate, reducing the number of steps/batches, and other methods that cause the clients to do less "work" per communication round may reduce the fluctuation.
I have a dataset of about a 100 numeric values. I have set the learning rate of my neural network to 0.0001. I have successfully trained it on the dataset for over 1 million times. But my question is that what is the effect of very low learning rates in the neural networks ?
Low learning rate mainly implies slow convergence: you're moving down the loss function with smaller steps (the step size is the learning rate).
If your function is convex this is not a problem, you will wait more but you'll reach a good solution.
If, as in the case of deep neural networks, your function is not convex than a low learning rate could lead to the reaching of a "good" optimum that's not the best one (getting stuck in a local minimum, without making steps as big as required to jump out of it).
That's why there are different optimization algorithms that are adaptive: such algorithm, like ADAM, RMSProp, ... have different learning rates for each weight in the network (every single learning rate starts from the same value). In this way, the optimization algorithm can work on every single parameter independently with the aim of finding a better solution (and letting the chose of the initial learning rate less critical)
I am trying to define the proper definition of the bad learning rate in the neural network as follows:
Bad learning rate in the neural network is when you assigned the learning rate too low or too high, with too low learning rate the network would take too much time to train but with too high learning rate the network would change too quick which might result in output.
Any suggestion would be much appreciated.
I believe that an efficient learning rate(alpha) depends on the data. The points that you have mentioned are absolutely correct with regards to inefficient learning rates. So, there is no hard and fast rule for selecting alpha. Let me enumerate the steps that I take while deciding alpha :
You would obviously need a big alpha so that your model learns quickly
Also take note that big alpha can lead to overshooting the minima and hence your hypothesis will not converge
To take care of this, you can go for learning rate decay. This decreases your learning rate as you approach minima and slow down learning so that your model doesn't overshoot.
Couple of ways to do this:
Step decay
exponential decay
linear decay
You can select either and then train your model. Having said that, let me point out that it still requires some trial and error from your side until you get the optimum result.
I was trying to train an emotion recognition model on the fer2013 dataset using the architecture proposed in this paper
The paper uses different dataset than mine so I did some modifications on on the stride and filter size.
After a couple hours of training, accuracy on both training and test set suddenly drops.
After that the accuracy just stay around 0.1-0.2 for both set, never improve anymore.
Does anybody know about this phenomenon?
In any neural network training, if both accuracies i.e. training and validation improves at first and then starts decreasing, it is a sign that your network is failing to converge. More appropriately, your optimizer has started overshooting.
One most likely reason for this could be high learning rate. Reduce your learning rate and then check your example again. Also, in your linked paper, (at least in first glimpse), I couldn't see learning rate mentioned. Since your data is different from the paper's, same learning rate might not work as well.
Is it worth to change learning rate after certain conditions are met? And how and why to do it? For example net will start with high learning rate and after squared error is low enough learning rate will drop for better precision or learning rate should increase to jump-out of local minima?. Wouldn't it cause over-fitting? And what about momentum?
Usually you should start with a high learning rate and a low momentum. Then you decrease the learning rate over time and increase the momentum. The idea is to allow more exploration at the beginning of the learning and force convergence at the end of the learning. Usually you should look at the training error to set up your learning schedule: if it got stuck, i.e. the error does not change, it is time to decrease your learning rate.