Related
I was trying to implement Linear Regression in Octave 5.1.0 on a data set relating the GRE score to the probability of Admission.
The data set is of the sort,
337 0.92
324 0.76
316 0.72
322 0.8
. . .
My main Program.m file looks like,
% read the data
data = load('Admission_Predict.txt');
% initiate variables
x = data(:,1);
y = data(:,2);
m = length(y);
theta = zeros(2,1);
alpha = 0.01;
iters = 1500;
J_hist = zeros(iters,1);
% plot data
subplot(1,2,1);
plot(x,y,'rx','MarkerSize', 10);
title('training data');
% compute cost function
x = [ones(m,1), (data(:,1) ./ 300)]; % feature scaling
J = computeCost(x,y,theta);
% run gradient descent
[theta, J_hist] = gradientDescent(x,y,theta,alpha,iters);
hold on;
subplot(1,2,1);
plot((x(:,2) .* 300), (x*theta),'-');
xlabel('GRE score');
ylabel('Probability');
hold off;
subplot (1,2,2);
plot(1:iters, J_hist, '-b');
xlabel('no: of iteration');
ylabel('Cost function');
computeCost.m looks like,
function J = computeCost(x,y,theta)
m = length(y);
h = x * theta;
J = (1/(2*m))*sum((h-y) .^ 2);
endfunction
and gradientDescent.m looks like,
function [theta, J_hist] = gradientDescent(x,y,theta,alpha,iters)
m = length(y);
J_hist = zeros(iters,1);
for i=1:iters
diff = (x*theta - y);
theta = theta - (alpha * (1/(m))) * (x' * diff);
J_hist(i) = computeCost(x,y,theta);
endfor
endfunction
The graphs plotted then looks like this,
which you can see, doesn't feel right even though my Cost function seems to be minimized.
Can someone please tell me if this is right? If not, what am I doing wrong?
The easiest way to check whether your implementation is correct is to compare with a validated implementation of linear regression. I suggest using an alternative implementation approach like the one suggested here, and then comparing your results. If the fits match, then this is the best linear fit to your data and if they don't match, then there may be something wrong in your implementation.
Suppose I have an image A, I applied Gaussian Blur on it with Sigam=3 So I got another Image B. Is there a way to know the applied sigma if A,B is given?
Further clarification:
Image A:
Image B:
I want to write a function that take A,B and return Sigma:
double get_sigma(cv::Mat const& A,cv::Mat const& B);
Any suggestions?
EDIT1: The suggested approach doesn't work in practice in its original form(i.e. using only 9 equations for a 3 x 3 kernel), and I realized this later. See EDIT1 below for an explanation and EDIT2 for a method that works.
EDIT2: As suggested by Humam, I used the Least Squares Estimate (LSE) to find the coefficients.
I think you can estimate the filter kernel by solving a linear system of equations in this case. A linear filter weighs the pixels in a window by its coefficients, then take their sum and assign this value to the center pixel of the window in the result image. So, for a 3 x 3 filter like
the resulting pixel value in the filtered image
result_pix_value = h11 * a(y, x) + h12 * a(y, x+1) + h13 * a(y, x+2) +
h21 * a(y+1, x) + h22 * a(y+1, x+1) + h23 * a(y+1, x+2) +
h31 * a(y+2, x) + h32 * a(y+2, x+1) + h33 * a(y+2, x+2)
where a's are the pixel values within the window in the original image. Here, for the 3 x 3 filter you have 9 unknowns, so you need 9 equations. You can obtain those 9 equations using 9 pixels in the resulting image. Then you can form an Ax = b system and solve for x to obtain the filter coefficients. With the coefficients available, I think you can find the sigma.
In the following example I'm using non-overlapping windows as shown to obtain the equations.
You don't have to know the size of the filter. If you use a larger size, the coefficients that are not relevant will be close to zero.
Your result image size is different than the input image, so i didn't use that image for following calculation. I use your input image and apply my own filter.
I tested this in Octave. You can quickly run it if you have Octave/Matlab. For Octave, you need to load the image package.
I'm using the following kernel to blur the image:
h =
0.10963 0.11184 0.10963
0.11184 0.11410 0.11184
0.10963 0.11184 0.10963
When I estimate it using a window size 5, I get the following. As I said, the coefficients that are not relevant are close to zero.
g =
9.5787e-015 -3.1508e-014 1.2974e-015 -3.4897e-015 1.2739e-014
-3.7248e-014 1.0963e-001 1.1184e-001 1.0963e-001 1.8418e-015
4.1825e-014 1.1184e-001 1.1410e-001 1.1184e-001 -7.3554e-014
-2.4861e-014 1.0963e-001 1.1184e-001 1.0963e-001 9.7664e-014
1.3692e-014 4.6182e-016 -2.9215e-014 3.1305e-014 -4.4875e-014
EDIT1:
First of all, my apologies.
This approach doesn't really work in the practice. I've used the filt = conv2(a, h, 'same'); in the code. The resulting image data type in this case is double, whereas in the actual image the data type is usually uint8, so there's loss of information, which we can think of as noise. I simulated this with the minor modification filt = floor(conv2(a, h, 'same'));, and then I don't get the expected results.
The sampling approach is not ideal, because it's possible that it results in a degenerated system. Better approach is to use random sampling, avoiding the borders and making sure the entries in the b vector are unique. In the ideal case, as in my code, we are making sure the system Ax = b has a unique solution this way.
One approach would be to reformulate this as Mv = 0 system and try to minimize the squared norm of Mv under the constraint squared-norm v = 1, which we can solve using SVD. I could be wrong here, and I haven't tried this.
Another approach is to use the symmetry of the Gaussian kernel. Then a 3x3 kernel will have only 3 unknowns instead of 9. I think, this way we impose additional constraints on v of the above paragraph.
I'll try these out and post the results, even if I don't get the expected results.
EDIT2:
Using the LSE, we can find the filter coefficients as pinv(A'A)A'b. For completion, I'm adding a simple (and slow) LSE code.
Initial Octave Code:
clear all
im = double(imread('I2vxD.png'));
k = 5;
r = floor(k/2);
a = im(:, :, 1); % take the red channel
h = fspecial('gaussian', [3 3], 5); % filter with a 3x3 gaussian
filt = conv2(a, h, 'same');
% use non-overlapping windows to for the Ax = b syatem
% NOTE: boundry error checking isn't performed in the code below
s = floor(size(a)/2);
y = s(1);
x = s(2);
w = k*k;
y1 = s(1)-floor(w/2) + r;
y2 = s(1)+floor(w/2);
x1 = s(2)-floor(w/2) + r;
x2 = s(2)+floor(w/2);
b = [];
A = [];
for y = y1:k:y2
for x = x1:k:x2
b = [b; filt(y, x)];
f = a(y-r:y+r, x-r:x+r);
A = [A; f(:)'];
end
end
% estimated filter kernel
g = reshape(A\b, k, k)
LSE method:
clear all
im = double(imread('I2vxD.png'));
k = 5;
r = floor(k/2);
a = im(:, :, 1); % take the red channel
h = fspecial('gaussian', [3 3], 5); % filter with a 3x3 gaussian
filt = floor(conv2(a, h, 'same'));
s = size(a);
y1 = r+2; y2 = s(1)-r-2;
x1 = r+2; x2 = s(2)-r-2;
b = [];
A = [];
for y = y1:2:y2
for x = x1:2:x2
b = [b; filt(y, x)];
f = a(y-r:y+r, x-r:x+r);
f = f(:)';
A = [A; f];
end
end
g = reshape(A\b, k, k) % A\b returns the least squares solution
%g = reshape(pinv(A'*A)*A'*b, k, k)
My goal is to create program on octave that loads audio file (wav, flac), calculates its mfcc features and serve them as output. The problem is that I do not have much experience with octave and cannot get octave load the audio file and that is why I am not sure if the extraction algorithms is correct. Is there simple way of loading the file and getting its features?
You can run mfcc code from RASTAMAT in octave, you only need to fix few things, the fixed version is available for download here.
The changes are to properly set windows in powspec.m
WINDOW = hanning(winpts);
and to fix the bug in specgram function which is not compatible with Matlab.
Check out Octave functions for calculating MFCC at https://github.com/jagdish7908/mfcc-octave
For a detailed theory on steps to compute MFCC, refer http://practicalcryptography.com/miscellaneous/machine-learning/guide-mel-frequency-cepstral-coefficients-mfccs/
function frame = create_frames(y, Fs, Fsize, Fstep)
N = length(y);
% divide the signal into frames with overlap = framestep
samplesPerFrame = floor(Fs*Fsize);
samplesPerFramestep = floor(Fs*Fstep);
i = 1;
frame = [];
while(i <= N-samplesPerFrame)
frame = [frame y(i:(i+samplesPerFrame-1))];
i = i+samplesPerFramestep;
endwhile
return
endfunction
function ans = hz2mel(f)
ans = 1125*log(1+f/700);
return
endfunction
function ans = mel2hz(f)
ans = 700*(exp(f/1125) - 1);
return
endfunction
function bank = melbank(n, min, max, sr)
% n = number of banks
% min = min frequency in hertz
% max = max frequency in hertz
% convert the min and max freq in mel scale
NFFT = 512;
% figure out bin value of min and max freq
minBin = floor((NFFT)*min/(sr/2));
maxBin = floor((NFFT)*max/(sr/2));
% convert the min, max in mel scale
min_mel = hz2mel(min);
max_mel = hz2mel(max);
m = [min_mel:(max_mel-min_mel)/(n+2-1):max_mel];
%disp(m);
h = mel2hz(m);
% replace frequencies in h with thier respective bin values
fbin = floor((NFFT)*h/(sr/2));
%disp(h);
% create triangular melfilter vectors
H = zeros(NFFT,n);
for vect = 2:n+1
for k = minBin:maxBin
if k >= fbin(vect-1) && k <= fbin(vect)
H(k,vect) = (k-fbin(vect-1))/(fbin(vect)-fbin(vect-1));
elseif k >= fbin(vect) && k <= fbin(vect+1)
H(k,vect) = (fbin(vect+1) - k)/(fbin(vect+1)-fbin(vect));
endif
endfor
endfor
bank = H;
return
endfunction
clc;
clear all;
close all;
pkg load signal;
% record audio
Fs = 44100;
y = record(3,44100);
% OR %
% Load existing file
%[y, Fs] = wavread('../FILE_PATH/');
%y = y(44100:2*44100);
% create mel filterbanks
minFreq = 500; % minimum cutoff frequency in Hz
maxFreq = 10000; % maximum cutoff frequency in Hz
% melbank(number_of_banks, minFreq, mazFreq, sampling_rate)
foo = melbank(30,minFreq,maxFreq,Fs);
% create frames
frames = create_frames(y, Fs, 0.025, 0.010);
% calculate periodogram of each frame
NF = length(frames(1,:));
[P,F] = periodogram(frames(:,1),[], 1024, Fs);
% apply mel filters to the power spectra
P = foo.*P(1:512);
% sum the energy in each filter and take the logarithm
P = log(sum(P));
% take the DCT of the log filterbank energies
% discard the first coeff 'cause it'll be -Inf after taking log
L = length(P);
P = dct(P(2:L));
PXX = P;
for i = 2:NF
P = periodogram(frames(:,i),[], 1024, Fs);
% apply mel filters to the power spectra
P = foo.*P(1:512);
% sum the energy in each filter and take the logarithm
P = log(sum(P));
% take the DCT of the log filterbank energies
% discard the first coeff 'cause it'll be -Inf after taking log
P = dct(P(2:L));
% coeffients are stacked row wise for each frame
PXX = [PXX; P];
endfor
% stack the coeffients column wise
PXX = PXX';
plot(PXX);
I'm in the second week of Professor Andrew Ng's Machine Learning course through Coursera. We're working on linear regression and right now I'm dealing with coding the cost function.
The code I've written solves the problem correctly but does not pass the submission process and fails the unit test because I have hard coded the values of theta and not allowed for more than two values for theta.
Here's the code I've got so far
function J = computeCost(X, y, theta)
m = length(y);
J = 0;
for i = 1:m,
h = theta(1) + theta(2) * X(i)
a = h - y(i);
b = a^2;
J = J + b;
end;
J = J * (1 / (2 * m));
end
the unit test is
computeCost( [1 2 3; 1 3 4; 1 4 5; 1 5 6], [7;6;5;4], [0.1;0.2;0.3])
and should produce ans = 7.0175
So I need to add another for loop to iterate over theta, therefore allowing for any number of values for theta, but I'll be damned if I can wrap my head around how/where.
Can anyone suggest a way I can allow for any number of values for theta within this function?
If you need more information to understand what I'm trying to ask, I will try my best to provide it.
You can use vectorize of operations in Octave/Matlab.
Iterate over entire vector - it is really bad idea, if your programm language let you vectorize operations.
R, Octave, Matlab, Python (numpy) allow this operation.
For example, you can get scalar production, if theta = (t0, t1, t2, t3) and X = (x0, x1, x2, x3) in the next way:
theta * X' = (t0, t1, t2, t3) * (x0, x1, x2, x3)' = t0*x0 + t1*x1 + t2*x2 + t3*x3
Result will be scalar.
For example, you can vectorize h in your code in the next way:
H = (theta'*X')';
S = sum((H - y) .^ 2);
J = S / (2*m);
Above answer is perfect but you can also do
H = (X*theta);
S = sum((H - y) .^ 2);
J = S / (2*m);
Rather than computing
(theta' * X')'
and then taking the transpose you can directly calculate
(X * theta)
It works perfectly.
The below line return the required 32.07 cost value while we run computeCost once using θ initialized to zeros:
J = (1/(2*m)) * (sum(((X * theta) - y).^2));
and is similar to the original formulas that is given below.
It can be also done in a line-
m- # training sets
J=(1/(2*m)) * ((((X * theta) - y).^2)'* ones(m,1));
J = sum(((X*theta)-y).^2)/(2*m);
ans = 32.073
Above answer is perfect,I thought the problem deeply for a day and still unfamiliar with Octave,so,Just study together!
If you want to use only matrix, so:
temp = (X * theta - y); % h(x) - y
J = ((temp')*temp)/(2 * m);
clear temp;
This would work just fine for you -
J = sum((X*theta - y).^2)*(1/(2*m))
This directly follows from the Cost Function Equation
Python code for the same :
def computeCost(X, y, theta):
m = y.size # number of training examples
J = 0
H = (X.dot(theta))
S = sum((H - y)**2);
J = S / (2*m);
return J
function J = computeCost(X, y, theta)
m = length(y);
J = 0;
% Hypothesis h(x)
h = X * theta;
% Error function (h(x) - y) ^ 2
squaredError = (h-y).^2;
% Cost function
J = sum(squaredError)/(2*m);
end
I think we needed to use iteration for much general solution for cost rather one iteration, also the result shows in the PDF 32.07 may not be correct answer that grader is looking for reason being its a one case out of many training data.
I think it should loop through like this
for i in 1:iteration
theta = theta - alpha*(1/m)(theta'*x-y)*x
j = (1/(2*m))(theta'*x-y)^2
I'm new to F# and in an effort to learn, thought it would be fun to implement a clustering algorithm.
I have an input list of lists that I need to iterate over. For each of these input vectors I need to apply a function that updates the weights and returns a list of lists (weight matrix). I can do that part via the newMatrix function. The problem is, I need to use the updated weight matrix in the next iteration, and I'm lost as to how to do this. Here's the important parts, some functions left out for brevity.
let inputList = [[1; 1; 0; 0]; [0; 0; 0; 1]; [1; 0; 0; 0]; [0; 0; 1; 1;]]
let weights = [[.2; .6; .5; .9]; [.8; .4; .7; .3]]
let newMatrix xi matrix =
List.map2( fun w wi ->
if wi = (yiIndex xi) then (newWeights xi)
else w) matrix [0..matrix.Length-1]
printfn "%A" (newMatrix inputList.Head weights)
> >
[[0.2; 0.6; 0.5; 0.9]; [0.92; 0.76; 0.28; 0.32]]
So my question is, how do I iterate over inputList calculating newMatrix for each inputVector using the previous newMatrix result?
Edit: added psuedo algorithm:
for input vector 1
given weight matrix calculate new weight matrix
return weight matirx prime
for input vector 2
given weight matrix prime calculate new weight matrix
and so on...
...
Aside: I'm implementing a Kohonen SOM algorithm fom this book.
If you just started learning F#, then it may be useful to try implementing this explicitly using recursion first. As Ankur points out, this particular recursive pattern is captured by List.fold, but it is quite useful to understand how List.fold actually works. So, the explicit version would look like this:
// Takes vectors to be processed and an initial list of weights.
// The result is an adapted list of weights.
let rec processVectors weights vectors =
match vectors with
| [] ->
// If 'vectors' is empty list, we're done and we just return current weights
weights
| head::tail ->
// We got a vector 'head' and remaining vectors 'tail'
// Adapt the weights using the current vector...
let weights2 = newweights weights head
// and then adapt weights using the remaining vectors (recursively)
processVectors weights2 tail
This is essentially what List.fold does, but it may be easier to understand it if you see the code written like this (the List.fold function hides the recursive processing, so the lambda function used as an argument is just the function that calculates new weights).
Aside, I don't quite understand your newMatrix function. Can you give more details about that? Generally, when working with lists you don't need to use indexing and it seems that you're doing something that requires accessing elements at a specific index. There may be a better way to write that....
I guess you are looking for List.fold.
Something like:
let inputList = [[1; 1; 0; 0]; [0; 0; 0; 1]; [1; 0; 0; 0]; [0; 0; 1; 1;]]
let weights = [[0.2; 0.6; 0.5; 0.9]; [0.8; 0.4; 0.7; 0.3]]
let newWeights w values = w //Fake method which returns old weight as it is
inputList |> List.fold (newWeights) weights
NOTE: The newWeights function in this case is taking weights and input vector and returns new weights
Or may be a List.scan in case you also need the intermediate calculated weights
let inputList = [[1; 1; 0; 0]; [0; 0; 0; 1]; [1; 0; 0; 0]; [0; 0; 1; 1;]]
let weights = [[0.2; 0.6; 0.5; 0.9]; [0.8; 0.4; 0.7; 0.3]]
let newWeights w values = w
inputList |> List.scan (newWeights) weights