Gradient Descent produces incorrect Thetas in octave - machine-learning

Question

I'm trying out a prediction algorithm using polynomial regression of the form h(x) = theta0 + theta1 * x1 + theta2 * x2, where x2=x1^2
I'm calculating the thetas with two methods, to compare the results: Normal Equation Vs. Gradient Decent. Then I plot the regression line for both methods for scores from 65 to 100, to see how it fits with my data.
When calculating thetas using Normal Equation, all seems to be working as expected. In the graph below, "x" is the actual scores, and "o" is the predicted scores.
However when calculating thetas using Gradient Decent, the resulting regression line does not fit my data. It looks like this:
While minimizing my Cost Function, I'm plotting the Gradient Descent iterations over J, to confirm that values converge. This seems to be working correctly:
Here's my code:
function [theta_normalEq, theta_gradientDesc] = a1_LinearRegression()
clear();
% suppose you want to fit a model of the form h(x) = theta0 + theta1 * x1 + theta2 * x2
% where x1 is the midterm score and x2 is (midterm score)^2
midTerm = [89; 72; 94; 69]; % mid-term Exam scores
midTerm2 = midTerm .^2; % same as above but each element squared (. refers to "each element". If 'dot' was not there, ^2 alone would mean matrix multiplications
X = [midTerm midTerm2]; % concatinate the two vectors into a single matrix
y = [96; 74; 87; 78]; %final Exam scores
% Method A:
% calculate theta (bias for each independent variable) using Normal Equation
% This works in some cases only (see comments in corresponding function below)
theta_normalEq = normalEquation(X, y);
% Method B:
% Use Gradient Descent
theta_gradientDesc = gradientDescent(X, y, 1.3, 60);
% plot regression line to see visually how it fits with our data
plotRegressionLine(midTerm, y, X, theta_gradientDesc);
% clear unneeded variables for a tidy output window
clear ('midTerm', 'midTerm2');
endfunction
% plots a regression line to see visually how it fits with our data
function plotRegressionLine(midTerm, y, X, theta)
% Our X matrix is n-long, but our theta is n+1 (remember we are modeling h(x) = theta0 + theta1 * x1 + theta2 * x2)
% Therefore we will introduce an X0 and set it to x0 = 1 for all values of i, so that we can do matrix operations with theta and X.
% This makes the two vectors 'theta' and x(i) match each other element-wise (that is, have the same number of elements: n+1).
X0 = ones(rows(X),1);
X = [X0 X]; % concatination; X had 2 columns, now it has 3. The very first column now consists of 'ones'
clear ('X0'); % just clears the variable
% with our thetas calculated, we can now plug them in our original model to make predictions
% model form: h(x) = theta0 + theta1 * x1 + theta2 * x2
% vectorized version: h(x) = X * theta
y_predicted = X * theta;
% let's also calculate the poits for all possible scores, to draw a regression line
scoreMin = 65;
scoreMax = 100;
step = 0.1;
scores = (scoreMin: step: scoreMax)';
scoresX = [ones(rows(scores),1) scores scores.^2];
scoresY_predicted = scoresX * theta;
% plot
figure 2;
clf;
hold on;
plot(midTerm, y, "x"); % draws our actual data points
plot(midTerm, y_predicted, "or"); % draws our predicted data points
plot(scores, scoresY_predicted, "r"); % draws our calculated regression line
hold off;
endfunction
% Performs gradient descent to learn theta. Updates theta by taking num_iters
%
% X = matrix of independent variables (e.g., size of house, number of bedrooms, number of bathrooms, etc)
% y = vector of dependent variables (e.g., cost of house)
% alpha = the rate of learning
% number of iterations to try finding the optimum theta
%
% Start by trying out a random alpha, like 0.1 or 1.
% If alpha is too small, it will take too long to minimize J and see values converging (too many iterations)
% If alpha is too large, we will overshoot the function minimum and values will start increasing again
% Ideally we want as large an alpha to get enough resolution to discover the function minimum with as few iterations as possible, without overshooting the minimum
%
% We also want a numner of iterations that are enough, but not too many. Depending on our problem and data, this can be from 30 to 300 to 3000 to 3 million, or more.
% In practice, we plot J against number of iterations as we go along the loop, to discover experimentally the optimal values for 'alpha' and 'num_iters'
% The graph we are looking for looks like a hokey stick of reducing values, that flattens horizontally. The J no longer reduces (the flat horizontal part), we have converged.
%
function theta = gradientDescent(X, y, alpha, num_iters)
% NORMALIZE FEATURES
% We can speed up gradient descent by having each of our input values in roughly the same range
% This is because θ will descend quickly on small ranges and slowly on large ranges, and so will oscillate inefficiently down to the optimum when the variables are very uneven.
% The way to prevent this is to modify the ranges of our input variables so that they are all roughly the same
% zscore() normalizes each feature (each column) independently, which is what we want: (value - mean of values for that column) / standard deviation of that column
X = zscore(X);
y = zscore(y);
% Our X matrix is n-long, but our theta is n+1 (remember we are modeling h(x) = theta0 + theta1 * x1 + theta2 * x2)
% Therefore we will introduce an X0 and set it to x0 = 1 for all values of i, so that we can do matrix operations with theta and X.
% This makes the two vectors 'theta' and x(i) match each other element-wise (that is, have the same number of elements: n+1).
X0 = ones(rows(X),1);
X = [X0 X]; % concatination; X had 2 columns, now it has 3. The very first column now consists of 'ones'
clear ('X0'); % just clears the variable
% number of training examples
m = length(y);
% save the cost J in every iteration in order to plot J vs. num_iters and check for convergence
J_history = zeros(num_iters, 1);
% We start with a random set of thetas.
% Gradient Descent improves them at each iteration until values converge
% NOTE: do not use randomMatrix() to initialize. Rather, hard code random values so that they are identical at each run attempt,
% to help us experiment with different sets of 'alpha' & 'num_iters' until we discover their optimal values.
%theta = randomMatrix(columns(X), 1, 0, 1);
theta = [0;0;0];
for iter = 1:num_iters
h = X * theta;
stderr = h - y;
theta = theta - (alpha/m) * X' * stderr;
J_history(iter) = computeCost(X, y, theta);
endfor
% plot J vs. num_iters and check for convergence
xAxis = 1:1:num_iters; % create vector from 1 to num_iters with step 1
figure 1;
clf;
plot(xAxis, J_history);
endfunction
% These two functions give identical results, but maybe one runs faster than another
function J = computeCost(X, y, theta)
m = length(y); % number of training examples
J = 1/(2*m) * sum( ( X*theta - y) .^ 2);
endfunction
%
function J = computeCostVectorized(X, y, theta)
m = length(y); % number of training examples
J = 1/(2*m) * (X*theta - y)' * (X*theta - y);
endfunction
% alternative way of finding the optimum theta without iteration and without having to try different alphas (rate of learning)
% however this method can be slow in situations with a lot of features + large training set combos
% There is no need to do feature scaling with the normal equation!!!
%
% WARNING:
% X'* X may be noninvertible. The common causes are:
% > Redundant features, where two features are very closely related (i.e. they are linearly dependent)
% > Too many features (e.g. m ≤ n). In this case, delete some features or use "regularization" (to be explained in a later lesson)
%
% Solutions to the above problems include deleting a feature that is linearly dependent with another or deleting one or more features when there are too many features
function theta = normalEquation(X, y)
% Our X matrix is n-long, but our theta is n+1 (remember we are modeling h(x) = theta0 + theta1 * x1 + theta2 * x2)
% Therefore we will introduce an X0 and set it to x0 = 1 for all values of i, so that we can do matrix operations with theta and X.
% This makes the two vectors 'theta' and x(i) match each other element-wise (that is, have the same number of elements: n+1).
X0 = ones(rows(X),1);
X = [X0 X]; % concatination; X had 2 columns, now it has 3. The very first column now consists of 'ones'
clear ('X0'); % just clears the variable
Xt = X';
theta = pinv(Xt * X) * Xt * y;
endfunction
% returns a random matrix of the specified size
% if you don't care to specify mean and variance, just use 0 and 1 respectively (or just call 'randn(rows, columns)' directly)
function retVal = randomMatrix(rows, columns, mean, variance)
retVal = mean + sqrt(variance)*(randn(rows,columns));
endfunction