accuracy of neural network is decreasing - machine-learning

Question

I have made program of neural network in python with help of neuralnetworksanddeeplearning.com. In which i have randomly initialized hiddenLayer weight of (784,100) and outputLayer weight (100,10).Algorithm is working on minibatch based theory and regularization overfitting with mnist.pkl.gz data set. I am using minibatch of size 10, learning rate(eta)=3, regularization parameter=2.5 . After run program its accuracy increses and then decrease...... So please help me how can i make it better for get more accuracy. following are itirations of algorithm. Thanks in Advance..
>>> stochastic(training_data,10,20,hiddenW,outW,hiddenB,outB,3,test_data,2.5)
Epoch 0 correct data: 9100.0/10000
Total cost of test data [ 307.75991542]
Epoch 1 correct data: 9136.0/10000
Total cost of test data [ 260.61199829]
Epoch 2 correct data: 9233.0/10000
Total cost of test data [ 244.9429907]
Epoch 3 correct data: 9149.0/10000
Total cost of test data [ 237.08391208]
Epoch 4 correct data: 9012.0/10000
Total cost of test data [ 227.14709858]
Epoch 5 correct data: 8714.0/10000
Total cost of test data [ 215.23668711]
Epoch 6 correct data: 8694.0/10000
Total cost of test data [ 201.79958056]
Epoch 7 correct data: 8224.0/10000
Total cost of test data [ 193.37639124]
Epoch 8 correct data: 7915.0/10000
Total cost of test data [ 183.83249811]
Epoch 9 correct data: 7615.0/10000
Total cost of test data [ 166.59631548]
# forward proppagation with with bais 3 para
def forward(weight,inp,b):
val=np.dot(weight.T,inp)+b
return val
# sigmoid function
def sigmoid(x):
val=1.0/(1.0+np.exp(-x))
return val
# Backpropagation for gradient check
def backpropagation(x,weight1,weight2,bais1,bais2,yTarget):
hh=forward(weight1,x,bais1)
hhout=sigmoid(hh)
oo=forward(weight2,hhout,bais2)
oout=sigmoid(oo)
ooe=-(yTarget-oout)*(oout*(1-oout))
hhe=np.dot(weight2,ooe)*(hhout*(1-hhout))
a2=np.dot(hhout,ooe.T)
a1=np.dot(x,hhe.T)
b1=hhe
b2=ooe
return a1,a2,b1,b2
def totalCost(data,weight1,weight2,bais1,bais2,lmbda):
m=len(data)
cost=0.0
for x,y in data:
hh=forward(weight1,x,bais1)
hhout=sigmoid(hh)
oo=forward(weight2,hhout,bais2)
oout=sigmoid(oo)
c=sum(-y*np.log(oout)-(1-y)*np.log(1-oout))
cost=cost+c/m
cost=cost+0.5*(lmbda/m)*(sum(map(sum,(weight1**2)))+sum(map(sum,(weight2**2))))
return cost
def stochastic(tdata,batch_size,epoch,w1,w2,b1,b2,eta,testdata,lmbda):
n=len(tdata)
for j in xrange(epoch):
random.shuffle(tdata)
mini_batches = [tdata[k:k+batch_size]for k in xrange(0, n, batch_size)]
for minibatch in mini_batches:
w1,w2,b1,b2=updateminibatch(minibatch,w1,w2,b1,b2,eta,lmbda)
print 'Epoch {0} correct data: {1}/{2}'.format(j,evaluate(testdata,w1,w2,b1,b2),len(testdata))
print 'Total cost of test data {0}'.format(totalCost(testdata,w1,w2,b1,b2,lmbda))
return w1,w2,b1,b2
def updateminibatch(data,w1,w2,b1,b2,eta,lmbda):
n=len(training_data)
q1=np.zeros(w1.shape)
q2=np.zeros(w2.shape)
q3=np.zeros(b1.shape)
q4=np.zeros(b2.shape)
for xin,yout in data:
delW1,delW2,delB1,delB2=backpropagation(xin,w1,w2,b1,b2,yout)
q1=q1+delW1
q2=q2+delW2
q3=q3+delB1
q4=q4+delB2
w1=(1-eta*(lmbda/n))*w1-(eta/len(data))*q1
w2=(1-eta*(lmbda/n))*w2-(eta/len(data))*q2
b1=b1-(eta/len(data))*q3
b2=b2-(eta/len(data))*q4
return w1,w2,b1,b2
def evaluate(testdata,w1,w2,b1,b2):
i=0
z=np.zeros(len(testdata))
for x,y in testdata:
h=forward(w1,x,b1)
hout=sigmoid(h)
o=forward(w2,hout,b2)
out=sigmoid(o)
p=np.argmax(out)
if (p==y):
a=int(p==y)
z[i]=a
i=i+1
return sum(z)

Answer 1

When you train a machine learning model, you must take care not to overfit your training data.
To understand if you are overfitting the data, is useful use 3 different sets of data during the training:
a training set, that you should use to train the model
a validation set, that you can use during the training to check if you are fitting the data accurately (clearly you don't have to use this set to train the model, but also as test during the training).
and a test set as final test of your model.
In particular is very useful the validation set. In fact if you are overfitting the data, is possible that you have a very good performance on the training set, but a low accuracy on this set. (-> in this case you model is too specialized on the training data, but probably has low accuracy in predict new data.)
So when the accuracy on the validation set start to decrease, is the moment to stop your training, because you have reached the best accuracy possible.
If you want to improve your model accuracy you could use more data for training, or, if you haven't or if accuracy doesn't increase, you should change you model, for example adding more layer in the neural network.