Back-propagation algorithm converging too quickly to poor results - machine-learning

Question

I'm trying to implement the back propagation algorithm for a multi layer feedforward neural network, but I'm having issues getting it to converge to good results. The reason being, the gradient descent gets stuck on a plate of the root mean squared error.
As you can see in the graph, there is very little change in the rms value for the first 70 epochs or so. Therefore the gradient descent things its found a minimum and stops. To fix this I set a requirement that the rms error must be below 0.3 in addition to the rate of change being below a given value. However, I don't think this is good as I believe there is something wrong with my implementation.
Below is the ruby code:
def train eta, criteria
rms = 1
old_rms = 0
rms_window = Array.new 20, 0
new_avg = 10
old_avg = 0
diff = 100
epoch = 0
#data[:training].shuffle!
while (diff > criteria || rms > 0.3) do
#while (diff > criteria) do
rms = 0
old_avg = new_avg
new_avg = 0
classification_error = 0
sample_num = 0
#data[:training].each_with_index do |s, s_i|
# Forward Propagation
inputs = [1, s[1], s[2]]
#hidden_layers.each_with_index do |hl, hl_i|
outputs = Array.new
# Bias Term
outputs << 1
# Compute the output for each neuron
hl.each do |p|
outputs << p.compute_output(inputs)
end
inputs = outputs
end
# Compute System Outputs
outputs = Array.new
#outputs.each do |p|
outputs << p.compute_output(inputs)
end
# Comput Errors
errors = Array.new
desired = #desired_values[s[0]-1]
#outputs.length.times do |x|
errors[x] = desired[x] - outputs[x]
rms += errors[x]**2
end
decision = outputs.each_with_index.max[1]
if decision+1 != s[0]
classification_error += 1
end
# Back Propagation
gradients = Array.new
local_gradient = Array.new
next_layer = Array.new
#outputs.each_with_index do |o, i|
local_gradient << errors[i] * o.activation_prime(o.output)
o.weights.length.times do |x|
o.weights[x] += eta * local_gradient[i] * o.inputs[x]
end
end
gradients << local_gradient
next_layer = #outputs
#hidden_layers.reverse_each do |hl|
local_gradient = Array.new
hl.each do |p|
gradient = 0
gradients.last.each_with_index do |g, i|
gradient += g * next_layer[i].weights[p.index+1]
end
gradient *= p.activation_prime(p.output)
local_gradient << gradient
p.weights.each_index do |x|
p.weights[x] += eta * gradient * p.inputs[x]
end
end
gradients << local_gradient
next_layer = hl
end
if s_i == 0
#puts "Epoch: #{epoch}\nOutputs: #{outputs}\nGradients:\n#{gradients[0]}\n#{gradients[1]}\n#{gradients[2]}\n\n"
#puts "Epoch #{epoch}\nError: #{errors}\nSE: #{rms}"
end
end
rms = Math::sqrt(rms / (#data[:training].length * 4))
rms_window[0] = rms
rms_window.rotate!
rms_window.each do |x|
new_avg += x
end
new_avg /= 20
diff = (new_avg - old_avg).abs
#rms << rms
epoch += 1
if classification_error == 0
break
end
#puts "RMS: #{rms}\tDiff: \t#{diff}\tClassification: #{classification_error}\n\n"
end
self.rms_plot "Plot"
self.grid_eval "Test", 250
end
The graph shown is for a 2-hidden layer network with 5 neurons in each hidden layer. There are 2 inputs and 4 outputs. Perhaps this is normal behavior, but something just seems off to me. Any help would be greatly appreciated.

Answer 1

There are many parameters that need to be tuned to get a multi-layer neural net to work. Based on my experiment, my first suggestions are:
1- give it a small set of synthesized data and run a baby project to see if the framework works.
2- Use a more convex cost function. There is no function that guarantees convexity, but there are many functions that are more convex that RMS.
3- Try scaling your input data in (-1,1) and output data in (0,1).
4- Try different values for learning rate.

Answer 2

In addition to whats already been said:
vary the range of the initial weights a little more (0 - 1 for example)
make sure your input data are properly normalised - i fell this can't be said often enough
vary the learning rate, start with sth like 0.05 and keep increasing/decreasing in small steps (if you find that changing your learning rate has a too extreme effect on the network's performance, then you may haven't normalised your input data appropriately)
shuffle the input data before every epoch
try using momentum (this essentially means, increase the learning rate while the gradient is steep, decrease if it becomes flatter), this often helps to jump over local optima
try using regularisation
experiment with the structure (add another hidden layer, increase the number of units in the hidden layer)