Related
In my neural network model, the test accuracy decrease in the iteration. I have check the learning rate and tune it smaller, but my test accuracy keep decreasing but not oscillating, so I think it is not the cause of the problem.
I use tempotron learning rule, and work on Iris dataset, which I use 100 training samples and 50 testing samples.
I have check my code, the test accuracy have increased at the beginning, so I think the learning rule do work on the weight.
But I can't figure out why the performance decrease after that.
Can someone have any ideas?
Thanks.
testing accuracy
for Iterate = 1:iteration %% Run 100 times
%% Test the correct rate each time
correct = 0;
for test_sample = 1:length(test)
% In each iteration, T = 100ms
for t = 1:T
for neuron = 1:neurons %% Response function for 48 neurons at time t
Response(neuron) = K(t,test(test_sample,neuron));
end
% Calculate PSP
for j = 1:3
V(j,t) = Response*weight(:,j) + V_rest;
end
end
%% find t_max: first index that V cross threshold
for j = 1:3
for timing = 1:T
if V(j,timing) >= threshold
t_max(j) = timing;
Max_state(j) = V(j,timing);
break;
end
end
V(j,t_max(j):end) = V(j,t_max(j)).*exp(-(Time(t_max(j):end)-Time(t_max(j)))/Tou_m);
end
[~,output_class] = min(t_max);
if output_class == test_target(test_sample)
correct = correct + 1;
end
end
correct_rate(Iterate) = correct/(length(test));
if Iterate > 1
if correct_rate(Iterate) < correct_rate(Iterate-1)
fprintf('Correct rate decrease\n');
%break;
end
end
%% Training
for samples = 1:size(InputSpike,1) %% Training samples for each iteration
% In each iteration, T = 100ms
for t = 1:T
for neuron = 1:neurons %% Response function for 48 neurons at time t
Response(neuron) = K(t,InputSpike(samples,neuron));
end
% Calculate PSP
for j = 1:3
V(j,t) = Response*weight(:,j) + V_rest;
end
end
%% find t_max: first index that V cross threshold
for j = 1:3
for timing = 1:T
if V(j,timing) >= threshold
t_max(j) = timing;
Max_state(j) = V(j,timing);
break;
end
end
V(j,t_max(j):end) = V(j,t_max(j)).*exp(-(Time(t_max(j):end)-
Time(t_max(j)))/Tou_m);
end
[~,output_class] = min(t_max);
%% weight modify when error occurs
if train_target(samples) ~= output_class
for j = 1:3
if j == train_target(samples) %% error in target neuron
if Max_state(j) < threshold %% if P+ error occurs
for i = 1:neurons
%% for all t_i < t_max
if InputSpike(samples,i) < t_max(j)
%% weight modified
weight(i,j) = weight(i,j) + ...
lr*K(t_max(j),InputSpike(samples,i));
end
end
end
elseif j ~= train_target(samples) %% error on other 2 output neurons
if Max_state(j) >= threshold %% if P- error occurs
for i = 1:neurons
%% for all t_i < t_max
if InputSpike(samples,i) < t_max(j)
%% weight modified
weight(i,j) = weight(i,j) - ...
lr*K(t_max(j),InputSpike(samples,i));
end
end
end
end
end
%% for neurons that fired but weaker than target neuron
elseif train_target(samples) == output_class
for j = 1:3
if j ~= train_target(samples) %% other 2 output neurons
if Max_state(j) >= threshold
for i = 1:neurons %% P- error occurs
%% for all t_i < t_max
if InputSpike(samples,i) < t_max(j)
%% weight modified
weight(i,j) = weight(i,j) - ...
lr*K(t_max(j),InputSpike(samples,i));
end
end
end
end
end
end
end
end
You should enlarge your training dataset to avoid overfitting. You can also try to increase your training epoches.
I'm trying to train a feed forward neural network for the first time in torch. Here's my dataset: http://ocw.mit.edu/courses/sloan-school-of-management/15-097-prediction-machine-learning-and-statistics-spring-2012/datasets/transfusion.csv
Here's the code (based, http://mdtux89.github.io/2015/12/11/torch-tutorial.html):
require 'nn'
mlp = nn.Sequential()
inputSize = 4
hiddenLayer1Size = 4
hiddenLayer2Size = 4
mlp:add(nn.Linear(inputSize,hiddenLayer1Size)) -- row, coulm
mlp:add(nn.Tanh())
mlp:add(nn.Linear(hiddenLayer1Size,hiddenLayer2Size))
mlp:add(nn.Tanh())
nclasses = 1
mlp:add(nn.Linear(hiddenLayer2Size,nclasses))
mlp:add(nn.LogSoftMax())
output = mlp:forward(torch.rand(1,4))
print(output)
-- TRAINING using inbuilt stochastic gradient descent, 2 params: network, criterian fun. --
LRate = 0.1
criterion = nn.ClassNLLCriterion()
trainer = nn.StochasticGradient(mlp, criterion)
trainer.learningRate = LRate
function string:splitAtCommas()
local sep, values = ",", {}
local pattern = string.format("([^%s]+)", sep)
self:gsub(pattern, function(c) values[#values+1] = c end)
return values
end
function loadData(dataFile)
local dataset,i = {},0
for line in io.lines(dataFile) do
local values = line:splitAtCommas()
local y = torch.Tensor(1)
y[1] = values[#values] -- the target class is the last number in the line
values[#values] = nil
local x = torch.Tensor(values) -- the input data is all the other numbers
dataset[i] = {x, y}
i = i + 1
end
function dataset:size() return (i - 1) end -- the requirement mentioned
return dataset
end
dataset = loadData("transfusion.csv")
trainer:train(dataset)
Here's the error report:
# StochasticGradient: training
/Users/drdre/torch/install/share/lua/5.1/nn/THNN.lua:109: Assertion `cur_target >= 0 && cur_target < n_classes' failed. at /Users/drdre/torch/extra/nn/lib/THNN/generic/ClassNLLCriterion.c:38
stack traceback:
[C]: in function 'v'
/Users/drdre/torch/install/share/lua/5.1/nn/THNN.lua:109: in function 'ClassNLLCriterion_updateOutput'
...dre/torch/install/share/lua/5.1/nn/ClassNLLCriterion.lua:41: in function 'forward'
...re/torch/install/share/lua/5.1/nn/StochasticGradient.lua:35: in function 'f'
[string "local f = function() return trainer:train(dat..."]:1: in main chunk
[C]: in function 'xpcall'
/Users/drdre/torch/install/share/lua/5.1/itorch/main.lua:209: in function </Users/drdre/torch/install/share/lua/5.1/itorch/main.lua:173>
/Users/drdre/torch/install/share/lua/5.1/lzmq/poller.lua:75: in function 'poll'
/Users/drdre/torch/install/share/lua/5.1/lzmq/impl/loop.lua:307: in function 'poll'
/Users/drdre/torch/install/share/lua/5.1/lzmq/impl/loop.lua:325: in function 'sleep_ex'
/Users/drdre/torch/install/share/lua/5.1/lzmq/impl/loop.lua:370: in function 'start'
/Users/drdre/torch/install/share/lua/5.1/itorch/main.lua:381: in main chunk
[C]: in function 'require'
(command line):1: in main chunk
[C]: at 0x0105e4cd10
Use nclasses = 2 and y[1] = values[#values] + 1. See the doc:
a desired output y (an integer 1 to n, in this case n = 2 classes)
require 'torch';
require 'nn';
require 'nnx';
mnist = require 'mnist';
fullset = mnist.traindataset()
testset = mnist.testdataset()
trainset = {
size = 50000,
data = fullset.data[{{1,50000}}]:double(),
label = fullset.label[{{1,50000}}]
}
validationset = {
size = 10000,
data = fullset.data[{{50001, 60000}}]:double(),
label = fullset.label[{{50001,60000}}]
}
-- MNIST Dataset has 28x28 images
model = nn.Sequential()
model:add(nn.SpatialConvolutionMM(1, 32, 5, 5)) -- 32x24x24
model:add(nn.ReLU())
model:add(nn.SpatialMaxPooling(3, 3, 3, 3)) -- 32x8x8
model:add(nn.SpatialConvolutionMM(32, 64, 5, 5)) -- 64x4x4
model:add(nn.Tanh())
model:add(nn.SpatialMaxPooling(2, 2, 2, 2)) -- 64x2x2
model:add(nn.Reshape(64*2*2))
model:add(nn.Linear(64*2*2, 200))
model:add(nn.Tanh())
model:add(nn.Linear(200, 10))
model:add(nn.LogSoftMax())
criterion = nn.ClassNLLCriterion()
x, dldx = model:getParameters() -- now x stores the trainable parameters and dldx stores the gradient wrt these params in the model above
sgd_params = {
learningRate = 1e-2,
learningRateDecay = 1e-4,
weightDecay = 1e-3,
momentum = 1e-4
}
step = function ( batchsize )
-- setting up variables
local count = 0
local current_loss = 0
local shuffle = torch.randperm(trainset.size)
-- setting default batchsize as 200
batchsize = batchsize or 200
-- setting inputs and targets for minibatches
for minibatch_number = 1, trainset.size, batchsize do
local size = math.min( trainset.size - minibatch_number + 1, batchsize )
local inputs = torch.Tensor(size, 28, 28)
local targets = torch.Tensor(size)
for index = 1, size do
inputs[index] = trainset.data[ shuffle[ index + minibatch_number ]]
targets[index] = trainset.label[ shuffle[ index + minibatch_number ] ]
end
-- defining feval function to return loss and gradients of loss w.r.t. params
feval = function( x_new )
--print ( "---------------------------------safe--------------------")
if x ~= x_new then x:copy(x_new) end
-- initializing gradParsams to zero
dldx:zero()
-- calculating loss and param gradients
local loss = criterion:forward( model.forward( inputs ), targets )
model:backward( inputs, criterion:backward( model.output, targets ) )
return loss, dldx
end
-- getting loss
-- optim returns x*, {fx} where x* is new set of params and {fx} is { loss } => fs[ 1 ] carries loss from feval
print(feval ~= nil and x ~= nil and sgd_params ~= nil)
_,fs = optim.sgd(feval, x, sgd_params)
count = count + 1
current_loss = current_loss + fs[ 1 ]
end
--returning avg loss over the minibatch
return current_loss / count
end
max_iters = 30
for i = 1 ,max_iters do
local loss = step()
print(string.format('Epoch: %d Current loss: %4f', i, loss))
end
I am new to torch and lua and I'm not able to find an error in the above code. Can anyone suggest a way to debug it?
The error:
/home/afroz/torch/install/bin/luajit: /home/afroz/test.lua:88: attempt to index global 'optim' (a nil value)
stack traceback:
/home/afroz/test.lua:88: in function 'step'
/home/afroz/test.lua:102: in main chunk
[C]: in function 'dofile'
...froz/torch/install/lib/luarocks/rocks/trepl/scm-1/bin/th:145: in main chunk
[C]: at 0x00406670
optim is not defined in the scope of your script. You try to call optim.sgd which of course results in the error you see.
Like nn, optim is a extension package to torch.
require 'torch';
require 'nn';
require 'nnx';
Remember those lines in the beginning of your script? They basically execute the definition of those packages.
Make sure optim is installed, then try to require it.
https://github.com/torch/optim
optim is not assigned anywhere in the script, so when the script references optim.sgd, its value is nil and you get the error you shown. You need to doublecheck the script to make sure the optim is assigned the correct value.
I'm working on a small Torch7/ Lua script to create and train a neural network, but I'm running into errors. Any ideas?
Here's my code:
require 'dp'
require 'csvigo'
require 'nn'
--[[hyperparameters]]--
opt = {
nHidden = 100, --number of hidden units
learningRate = 0.1, --training learning rate
momentum = 0.9, --momentum factor to use for training
maxOutNorm = 1, --maximum norm allowed for output neuron weights
batchSize = 128, --number of examples per mini-batch
maxTries = 100, --maximum number of epochs without reduction in validation error.
maxEpoch = 1 --maximum number of epochs of training
}
csv2tensor = require 'csv2tensor'
-- inputs, outputs = csv2tensor.load("/Users/robertgrzesik/NodeJS/csv_export.csv")
inputs = csv2tensor.load("/Users/robertgrzesik/NodeJS/csv_export.csv", {exclude={"positive", "negative", "neutral"}})
outputs = csv2tensor.load("/Users/robertgrzesik/NodeJS/csv_export.csv", {include={"positive", "negative", "neutral"}}) -- "positive", "negative", "neutral"
print("outputs: ", outputs)
print("inputs: ", inputs)
local dataset = {}
print("inputs:size(1)", inputs:size(1))
inputSize = inputs:size(1)
outputSize = outputs:size(1)
for i=1,inputSize do
dataset[i] = {inputs[i], outputs[i]}
end
dataset.size = function(self)
return inputSize
end
-- ======================================= --
-- Create NN
-- ======================================= --
print '[INFO] Creating NN..'
mlp = nn.Sequential(); -- make a multi-layer perceptron
inputs = inputSize; outputs = outputSize; HUs = 300; -- parameters
mlp:add(nn.Linear(inputs, HUs))
mlp:add(nn.Tanh())
mlp:add(nn.Linear(HUs, outputs))
-- ======================================= --
-- MSE and Training
-- ======================================= --
print '[INFO] MSE and train NN..'
criterion = nn.MSECriterion()
trainer = nn.StochasticGradient(mlp, criterion)
trainer.learningRate = 0.01
trainer:train(dataset)
Here's the error:
# StochasticGradient: training
/Users/robertgrzesik/torch/install/bin/luajit: .../robertgrzesik/torch/install/share/lua/5.1/nn/Linear.lua:37: size mismatch
stack traceback:
[C]: in function 'addmv'
.../robertgrzesik/torch/install/share/lua/5.1/nn/Linear.lua:37: in function 'updateOutput'
...ertgrzesik/torch/install/share/lua/5.1/nn/Sequential.lua:25: in function 'forward'
...ik/torch/install/share/lua/5.1/nn/StochasticGradient.lua:35: in function 'train'
/Users/robertgrzesik/Lua/async-master/tests/dp-test.lua:53: in main chunk
[C]: in function 'dofile'
...esik/torch/install/lib/luarocks/rocks/trepl/scm-1/bin/th:131: in main chunk
[C]: at 0x01028bc780
And here's a sample of my data:
positive,negative,basketball,neutral,the,be,and,of,a,in,to,have,it,I,for,that,he,you,with,on,do,this,they,at,who,if,her,people,take,your,like,our,new,because,woman,great,show,million,money,job,little,important,lose,include,rest,fight,perfect
0,0,0,1,3,0,1,0,0,0,1,0,0,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0
0,1,0,0,0,0,1,0,0,1,0,0,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0
Basically my aim is to create a deep neural network linking the frequency of words used in a sentence and tie it to the user rating it as either "positive", "negative" or "neutral" (my outputs, which are binary). Please also let me know if my thinking is correct on this.
Thank you!
Found the problem!
The issue was that I was giving the wrong sizes when creating the network. I was passing in "inputs:size(1)" instead of "inputs:size(2)". Here's the fix
mlp:add(nn.Linear(inputs:size(2), HUs))
mlp:add(nn.Tanh())
mlp:add(nn.Linear(HUs, outputs:size(2)))
Feel like I'm slowly starting to get the hang of Lua/ Torch! Score
I write JPEG compression in Scilab (equivalent of MATLAB) using function imdct. In this function is used function DCT from openCV and I don't know which equation is used in dct function.
lenna by imdct
lenna by my_function
You can see lenna by imdct which is internal function and lenna by my_function is my function in scilab.
I add my code in scilab
function vystup = dct_rovnice(vstup)
[M,N] = size(vstup)
for u=1:M
for v=1:N
cos_celkem = 0;
for m=1:M
for n=1:N
pom = double(vstup(m,n));
cos_citatel1 = cos(((2*m) * u * %pi)/(2*M));
cos_citatel2 = cos(((2*n) * v * %pi)/(2*N));
cos_celkem = cos_celkem + (pom * cos_citatel1 * cos_citatel2);
end
end
c_u = 0;
c_v = 0;
if u == 1 then
c_u = 1 / sqrt(2);
else
c_u = 1;
end
if v == 1 then
c_v = 1 / sqrt(2);
else
c_v = 1;
end
vystup(u,v) = (2/sqrt(n*m)) * c_u * c_v * cos_celkem;
end
end
endfunction
function vystup = dct_prevod(vstup)
Y = vstup(:,:,1);
Cb = vstup(:,:,2);
Cr = vstup(:,:,3);
[rows,columns]=size(vstup)
vystup = zeros(rows,columns,3)
for y=1:8:rows-7
for x=1:8:columns-7
blok_Y = Y(y:y+7,x:x+7)
blok_Cb = Cb(y:y+7,x:x+7)
blok_Cr = Cr(y:y+7,x:x+7)
blok_dct_Y = dct_rovnice(blok_Y)
blok_dct_Cb = dct_rovnice(blok_Cb)
blok_dct_Cr = dct_rovnice(blok_Cr)
vystup(y:y+7,x:x+7,1)= blok_dct_Y
vystup(y:y+7,x:x+7,2)= blok_dct_Cb
vystup(y:y+7,x:x+7,3)= blok_dct_Cr
end
end
vystup = uint8(vystup)
endfunction
You can see equation I used
EQUATION
The issue seems to be in the use of different normalization of the resulting coefficients.
The OpenCV library uses this equation for a forward transform (N=8, in your case):
The basis g is defined as
where
(Sorry for the ugly images, but SO does not provide any support for typesetting equations.)
Take care there are several definitions of the dct function (DCT-I, DCT-II, DCT-III and DCT-IV normalized and un-normmalized)
Moreover have you tried the Scilab builtin function dct (from FFTW) which can be applied straightforward to images.