Gradient descent values not correct - machine-learning

I'm attempting to implement gradient descent using code from :
Gradient Descent implementation in octave
I've amended code to following :
X = [1; 1; 1;]
y = [1; 0; 1;]
m = length(y);
X = [ones(m, 1), data(:,1)];
theta = zeros(2, 1);
iterations = 2000;
alpha = 0.001;
for iter = 1:iterations
theta = theta -((1/m) * ((X * theta) - y)' * X)' * alpha;
end
theta
Which gives following output :
X =
1
1
1
y =
1
0
1
theta =
0.32725
0.32725
theta is a 1x2 Matrix but should'nt it be 1x3 as the output (y) is 3x1 ?
So I should be able to multiply theta by the training example to make a prediction but cannot multiply x by theta as x is 1x3 and theta is 1x2?
Update :
%X = [1 1; 1 1; 1 1;]
%y = [1 1; 0 1; 1 1;]
X = [1 1 1; 1 1 1; 0 0 0;]
y = [1 1 1; 0 0 0; 1 1 1;]
m = length(y);
X = [ones(m, 1), X];
theta = zeros(4, 1);
theta
iterations = 2000;
alpha = 0.001;
for iter = 1:iterations
theta = theta -((1/m) * ((X * theta) - y)' * X)' * alpha;
end
%to make prediction
m = size(X, 1); % Number of training examples
p = zeros(m, 1);
htheta = sigmoid(X * theta);
p = htheta >= 0.5;

You are misinterpreting dimensions here. Your data consists of 3 points, each having a single dimension. Furthermore, you add a dummy dimension of 1s
X = [ones(m, 1), data(:,1)];
thus
octave:1> data = [1;2;3]
data =
1
2
3
octave:2> [ones(m, 1), data(:,1)]
ans =
1 1
1 2
1 3
and theta is your parametrization, which you should be able to apply through (this is not a code, but math notation)
h(x) = x1 * theta1 + theta0
thus your theta should have two dimensions. One is a weight for your dummy dimension (so called bias) and one for actual X dimension. If your X has K dimensions, theta would have K+1. Thus, after adding a dummy dimension matrices have following shapes:
X is 3x2
y is 3x1
theta is 2x1
so
X * theta is 3x1
the same as y

Related

Why my cost function is giving wrong answer?

I have written a code for the cost function and it is giving incorrect answer.
I have read the code many times but I cannot find the mistake.
Here is my code:-
function J = computeCost(X, y, theta)
m = length(y); % number of training examples
s = 0;
h = 0;
sq = 0;
J = 0;
for i = 1:m
h = theta' * X(i, :)';
sq = (h - y(i))^2;
s = s + sq;
end
J = (1/2*m) * s;
end
Example:-
computeCost( [1 2; 1 3; 1 4; 1 5], [7;6;5;4], [0.1;0.2] )
ans = 11.9450
Here the answer should be 11.9450 but my code is giving me this:-
ans = 191.12
I have checked the the matrix multiplication and the code is calculating it right.
It seems you misunderstood the operator evaluation order. In fact
1/2*m ~= 1/(2*m)
With this in mind it seems you're computing an average. Instead of reinventing the wheel it is usually a good idea to use the built in functions to do the job which results in a much clearer (and less error prone) implementation:
function J = computeCost(X, y, theta)
h = X * theta;
sq = (h - y).^2;
J = 1/2 * mean(sq);
end
computeCost( [1,2;1,3;1,4;1,5], [7;6;5;4], [0.1;0.2] )
% ans = 11.9450
Try it online!

My gradient descent is not giving the exact value

I have written gradient descent algorithm in Octave but it is not giving me the exact answer. The answer differs from one to two digits.
Here is my code:
function theta = gradientDescent(X, y, theta, alpha, num_iters)
m = length(y); % number of training examples
s = 0;
temp = theta;
for iter = 1:num_iters
for j = 1:size(theta, 1)
for i = 1:m
h = theta' * X(i, :)';
s = s + (h - y(i))*X(i, j);
end
s = s/m;
temp(j) = temp(j) - alpha * s;
end
theta = temp;
end
end
For:
theta = gradientDescent([1 5; 1 2; 1 4; 1 5],[1 6 4 2]',[0 0]',0.01,1000);
My gradient descent gives this:
4.93708
-0.50549
But it is expected to give this:
5.2148
-0.5733
Minor fixes :
Your variable s probably the delta is initialised incorrectly.
So it the temp variable probably the new theta
Incorrectly calculating the delta
Try with below changes.
function [theta, J_history] = gradientDescent(X, y, theta, alpha, num_iters)
m = length(y); % number of training examples
J_history = zeros(num_iters, 1);
temp = theta;
for iter = 1:num_iters
temp = zeros(length(theta), 1);
for j = 1:size(theta)
s = 0
for i = 1:m
s = s + (X(i, :)*theta - y(i)) * X(i, j);
end
end
s = s/m;
temp(j) = temp(j) - alpha * s;
end
theta = temp;
J_history(iter) = computeCost(X, y, theta);
end
end

decomposeProjectionMatrix gives unexpected result

I have the following projection matrix P:
-375 0 2000 262500
-375 2000 0 262500
-1 0 0 700
This projection matrix projects 3D points in mm on a detector in px (with 1px equals to 0.5mm) and is built from the intrinsic matrix K and the extrinsic matrix [R|t] (where R is a rotation matrix and t a translation vector) according the relation P = K [R|t].
2000 0 375 0 0 1 0
K = 0 2000 375 R = 0 1 0 t = 0
0 0 1 -1 0 0 700
For some reasons I need to decompose P back into these matrices. When I use decomposeProjectionMatrix I get this as a rotation matrix:
0 0 0
0 0 0
-1 0 0
Which doesn't look like a rotation matrix to me.
Moreover when I build back the projection matrix from the Open CV decomposition I get this matrix:
-375 0 0 262500
-375 0 0 262500
-1 0 0 700
Looks similar but it is not the same.
I'm wondering if I'm doing something wrong or if I'm unlucky and that was one of the rare cases where this function fails.
Note that I did the decomposition by myself and I get coherent results but I would rather use Open CV functions as much as possible.
The problem seems to be in the RQ decomposition used by decomposeProjectionMatrix.
Even though the first square of the matrix P is non singular, the RQDecomp3x3 function gives incorrect results:
0 0 375 0 0 0
R = 0 0 375 Q = 0 0 0
0 0 1 -1 0 0
So a work around is to use a homemade function (here written in Python) based on the section 2.2 of Peter Sturm's lectures:
def decomposeP(P):
import numpy as np
M = P[0:3,0:3]
Q = np.eye(3)[::-1]
P_b = Q # M # M.T # Q
K_h = Q # np.linalg.cholesky(P_b) # Q
K = K_h / K_h[2,2]
A = np.linalg.inv(K) # M
l = (1/np.linalg.det(A)) ** (1/3)
R = l * A
t = l * np.linalg.inv(K) # P[0:3,3]
return K, R, t
I use the anti-identity matrix Q to build the non conventional Cholesky decomposition U U* where U is upper triangular.
This method differs slightly from the Peter Sturm's one as we use the relation P = K[R|t] while in Peter Sturm's lectures the relation used is P = K[R|-Rt].
A C++ implementation using only Open CV is trickier as they don't really expose a function for Cholesky decompostion:
void chol(cv::Mat const& S, cv::Mat& L)
{
L = cv::Mat::zeros(S.rows, S.rows, cv::DataType<double>::type);
for (int i = 0; i < S.rows; ++i) {
for (int j = 0; j <= i ; ++j) {
double sum = 0;
for (int k = 0; k < j; ++k)
sum += L.at<double>(i,k) * L.at<double>(j,k);
L.at<double>(i,j) = (i == j) ? sqrt(S.at<double>(i,i) - sum) : (S.at<double>(i,j) - sum) / L.at<double>(j,j);
}
}
}
void decomposeP(cv::Mat const& P, cv::Mat& K, cv::Mat& R, cv::Mat& t)
{
cv::Mat M(3, 3, cv::DataType<double>::type);
for (int i = 0; i < 3; ++i)
for (int j = 0; j < 3; ++j)
M.at<double>(i, j) = P.at<double>(i ,j);
cv::Mat Q = cv::Mat::zeros(3, 3, cv::DataType<double>::type);
Q.at<double>(0, 2) = 1.0;
Q.at<double>(1, 1) = 1.0;
Q.at<double>(2, 0) = 1.0;
cv::Mat O = Q * M * M.t() * Q;
cv::Mat C;
chol(O, C);
cv::Mat B = Q * C * Q;
K = B / B.at<double>(2,2);
cv::Mat A = K.inv() * M;
double l = std::pow((1 / cv::determinant(A)), 1/3);
R = l * A;
cv::Mat p(3, 1, cv::DataType<double>::type);
for (int i = 0; i < 3; ++i)
p.at<double>(i, 0) = P.at<double>(i ,3);
t = l * K.inv() * p;
}

Convert cv::Vec4f line to cv::Vec2f

I have a pair of Cartesian coordinates that represent a line in an image. I would like to convert this line to polar form and draw it over the image.
e.g
cv::Vec4f line {10,20,60,70};
float x1 = line[0];
float y1 = line[1];
float x2 = line[2];
float y2 = line[3];
I want this line to be represented in cv::Vec2f form(rho,theta).
Taking care of rho & theta with all possible slopes.
Given are the image dimensions :: w and h;
w = image.cols
h = image.rows
How can I achieve this.
N.B: We can also assume that the line can be an extended one running across the image.
for (size_t i = 0; i < lines.size(); i++)
{
int x1 = lines[i][0];
int y1 = lines[i][1];
int x2 = lines[i][2];
int y2 = lines[i][3];
float d = sqrt(((y1-y2)*(y1-y2)) + ((x2-x1)*(x2-x1)) );
float rho = (y1*x2 - y2*x1)/d;
float theta = atan2(x2 - x1,y1-y2) ;
if(rho < 0){
theta *= -1;
rho *= -1;
}
linv2f.push_back(cv::Vec2f(rho,theta));
}
The above approach doesnt give me results when I plot the lines I dont get the lines that are overlapping their original vec4f form.
I use this to convert vec2f to vec4f for testing :
cv::Vec4f cvtVec2fLine(const cv::Vec2f& data, const cv::Mat& img)
{
float const rho = data[0];
float const theta = data[1];
cv::Point pt1,pt2;
if((theta < CV_PI/4. || theta > 3. * CV_PI/4.)){
pt1 = cv::Point(rho / std::cos(theta), 0);
pt2 = cv::Point( (rho - img.rows * std::sin(theta))/std::cos(theta), img.rows);
}else {
pt1 = cv::Point(0, rho / std::sin(theta));
pt2 = cv::Point(img.cols, (rho - img.cols * std::cos(theta))/std::sin(theta));
}
cv::Vec4f l;
l[0] = pt1.x;
l[1] = pt1.y;
l[2] = pt2.x;
l[3] = pt2.y;
return l;
}
rho-theta equation has form
x * Cos(Theta) + y * Sin(Theta) - Rho = 0
We want to represent equation 'by two points' into rho-theta form (page 92 in pdf here). If we have
x * A + y * B - C = 0
and need coefficients in trigonometric form, we can divide all equation by magnitude of (A,B) coefficient vector.
D = Length(A,B) = Math.Hypot(A,B)
x * A/D + y * B/D - C/D = 0
note that (A/D)^2 + (B/D)^2 = 1 - basic trigonometric equality, so we can consider A/D and B/D as cosine and sine of some angle theta.
Your line equation is
(y-y1) * (x2-x1) - (x-x1) * (y2-y1) = 0
or
x * (y1-y2) + y * (x2-x1) - (y1 * x2 - y2 * x1) = 0
let
D = Sqrt((y1-y2)^2 + (x2-x1)^2)
so
Theta = ArcTan2(x2-x1, y1-y2)
Rho = (y1 * x2 - y2 * x1) / D
edited
If Rho is negative, change sign of Rho and shift Theta by Pi
Example:
x1=1,y1=0, x2=0,y2=1
Theta = atan2(-1,-1)=-3*Pi/4
D=Sqrt(2)
Rho=-Sqrt(2)/2 negative =>
Rho = Sqrt(2)/2
Theta = Pi/4
Back substitutuon - find points of intersection with axes
0 * Sqrt(2)/2 + y0 * Sqrt(2)/2 - Sqrt(2)/2 = 0
x=0 y=1
x0 * Sqrt(2)/2 + 0 * Sqrt(2)/2 - Sqrt(2)/2 = 0
x=1 y=0

How can I get ellipse coefficient from fitEllipse function of OpenCV?

I want to extract the red ball from one picture and get the detected ellipse matrix in picture.
Here is my example:
I threshold the picture, find the contour of red ball by using findContour() function and use fitEllipse() to fit an ellipse.
But what I want is to get coefficient of this ellipse. Because the fitEllipse() return a rotation rectangle (RotatedRect), so I need to re-write this function.
One Ellipse can be expressed as Ax^2 + By^2 + Cxy + Dx + Ey + F = 0; So I want to get u=(A,B,C,D,E,F) or u=(A,B,C,D,E) if F is 1 (to construct an ellipse matrix).
I read the source code of fitEllipse(), there are totally three SVD process, I think I can get the above coefficients from the results of those three SVD process. But I am quite confused what does each result (variable cv::Mat x) of each SVD process represent and why there are three SVD here?
Here is this function:
cv::RotatedRect cv::fitEllipse( InputArray _points )
{
Mat points = _points.getMat();
int i, n = points.checkVector(2);
int depth = points.depth();
CV_Assert( n >= 0 && (depth == CV_32F || depth == CV_32S));
RotatedRect box;
if( n < 5 )
CV_Error( CV_StsBadSize, "There should be at least 5 points to fit the ellipse" );
// New fitellipse algorithm, contributed by Dr. Daniel Weiss
Point2f c(0,0);
double gfp[5], rp[5], t;
const double min_eps = 1e-8;
bool is_float = depth == CV_32F;
const Point* ptsi = points.ptr<Point>();
const Point2f* ptsf = points.ptr<Point2f>();
AutoBuffer<double> _Ad(n*5), _bd(n);
double *Ad = _Ad, *bd = _bd;
// first fit for parameters A - E
Mat A( n, 5, CV_64F, Ad );
Mat b( n, 1, CV_64F, bd );
Mat x( 5, 1, CV_64F, gfp );
for( i = 0; i < n; i++ )
{
Point2f p = is_float ? ptsf[i] : Point2f((float)ptsi[i].x, (float)ptsi[i].y);
c += p;
}
c.x /= n;
c.y /= n;
for( i = 0; i < n; i++ )
{
Point2f p = is_float ? ptsf[i] : Point2f((float)ptsi[i].x, (float)ptsi[i].y);
p -= c;
bd[i] = 10000.0; // 1.0?
Ad[i*5] = -(double)p.x * p.x; // A - C signs inverted as proposed by APP
Ad[i*5 + 1] = -(double)p.y * p.y;
Ad[i*5 + 2] = -(double)p.x * p.y;
Ad[i*5 + 3] = p.x;
Ad[i*5 + 4] = p.y;
}
solve(A, b, x, DECOMP_SVD);
// now use general-form parameters A - E to find the ellipse center:
// differentiate general form wrt x/y to get two equations for cx and cy
A = Mat( 2, 2, CV_64F, Ad );
b = Mat( 2, 1, CV_64F, bd );
x = Mat( 2, 1, CV_64F, rp );
Ad[0] = 2 * gfp[0];
Ad[1] = Ad[2] = gfp[2];
Ad[3] = 2 * gfp[1];
bd[0] = gfp[3];
bd[1] = gfp[4];
solve( A, b, x, DECOMP_SVD );
// re-fit for parameters A - C with those center coordinates
A = Mat( n, 3, CV_64F, Ad );
b = Mat( n, 1, CV_64F, bd );
x = Mat( 3, 1, CV_64F, gfp );
for( i = 0; i < n; i++ )
{
Point2f p = is_float ? ptsf[i] : Point2f((float)ptsi[i].x, (float)ptsi[i].y);
p -= c;
bd[i] = 1.0;
Ad[i * 3] = (p.x - rp[0]) * (p.x - rp[0]);
Ad[i * 3 + 1] = (p.y - rp[1]) * (p.y - rp[1]);
Ad[i * 3 + 2] = (p.x - rp[0]) * (p.y - rp[1]);
}
solve(A, b, x, DECOMP_SVD);
// store angle and radii
rp[4] = -0.5 * atan2(gfp[2], gfp[1] - gfp[0]); // convert from APP angle usage
if( fabs(gfp[2]) > min_eps )
t = gfp[2]/sin(-2.0 * rp[4]);
else // ellipse is rotated by an integer multiple of pi/2
t = gfp[1] - gfp[0];
rp[2] = fabs(gfp[0] + gfp[1] - t);
if( rp[2] > min_eps )
rp[2] = std::sqrt(2.0 / rp[2]);
rp[3] = fabs(gfp[0] + gfp[1] + t);
if( rp[3] > min_eps )
rp[3] = std::sqrt(2.0 / rp[3]);
box.center.x = (float)rp[0] + c.x;
box.center.y = (float)rp[1] + c.y;
box.size.width = (float)(rp[2]*2);
box.size.height = (float)(rp[3]*2);
if( box.size.width > box.size.height )
{
float tmp;
CV_SWAP( box.size.width, box.size.height, tmp );
box.angle = (float)(90 + rp[4]*180/CV_PI);
}
if( box.angle < -180 )
box.angle += 360;
if( box.angle > 360 )
box.angle -= 360;
return box;
}
The source code link: https://github.com/Itseez/opencv/blob/master/modules/imgproc/src/shapedescr.cpp
The function fitEllipse returns a RotatedRect that contains all the parameters of the ellipse.
An ellipse is defined by 5 parameters:
xc : x coordinate of the center
yc : y coordinate of the center
a : major semi-axis
b : minor semi-axis
theta : rotation angle
You can obtain these parameters like:
RotatedRect e = fitEllipse(points);
float xc = e.center.x;
float yc = e.center.y;
float a = e.size.width / 2; // width >= height
float b = e.size.height / 2;
float theta = e.angle; // in degrees
You can draw an ellipse with the function ellipse using the RotatedRect:
ellipse(image, e, Scalar(0,255,0));
or, equivalently using the ellipse parameters:
ellipse(res, Point(xc, yc), Size(a, b), theta, 0.0, 360.0, Scalar(0,255,0));
If you need the values of the coefficients of the implicit equation, you can do like (from Wikipedia):
So, you can get the parameters you need from the RotatedRect, and you don't need to change the function fitEllipse.
The solve function is used to solve linear systems or least-squares problems. Using the SVD decomposition method the system can be over-defined and/or the matrix src1 can be singular.
For more details on the algorithm, you can see the paper of Fitzgibbon that proposed this fit ellipse method.
Here is some code that worked for me which I based on the other responses on this thread.
def getConicCoeffFromEllipse(e):
# ellipse(Point(xc, yc),Size(a, b), theta)
xc = e[0][0]
yc = e[0][1]
a = e[1][0]/2
b = e[1][1]/2
theta = math.radians(e[2])
# See https://en.wikipedia.org/wiki/Ellipse
# Ax^2 + Bxy + Cy^2 + Dx + Ey + F = 0 is the equation
A = a*a*math.pow(math.sin(theta),2) + b*b*math.pow(math.cos(theta),2)
B = 2*(b*b - a*a)*math.sin(theta)*math.cos(theta)
C = a*a*math.pow(math.cos(theta),2) + b*b*math.pow(math.sin(theta),2)
D = -2*A*xc - B*yc
E = -B*xc - 2*C*yc
F = A*xc*xc + B*xc*yc + C*yc*yc - a*a*b*b
coef = np.array([A,B,C,D,E,F]) / F
return coef
def getConicMatrixFromCoeff(c):
C = np.array([[c[0], c[1]/2, c[3]/2], # [ a, b/2, d/2 ]
[c[1]/2, c[2], c[4]/2], # [b/2, c, e/2 ]
[c[3]/2, c[4]/2, c[5]]]) # [d/2], e/2, f ]
return C

Resources