completing the square in machine learning - machine-learning

Question

I have a question on completing the square in multivariate analysis and I am not able to solve it.

Answer 1

The hint made the job more difficult, otherwise it would just be a mechanical work expand the products, replace the definition for $\hat w$ and compare terms.
I had a hard time recognizing the correct substitutions, good exercise though.
It is important to notice that there is a type on the hint, the first term must be with lower case $x$, instead of $X$.
Solution
Prove that $||X' W - y||^2 = (w - \hat w)X' X (w - \hat w) + ||X' \hat w - y||^2$
where
$\hat w = (X' X)^{-1} X y$
hint:
$x^T M x - 2b'x = (x - M^{-1}b)'M(x - M^{-1}b) - b'M^{-1}b$
for any vectors $x$ and $b$, and symmetric matrix $M$
Express the norms as matrix products $||z||^2 = z'z$
$(X' w - y)'(X' W - y) = (w - \hat w)X' X (w - \hat w) + (X' \hat w - y)'(X' \hat w - y)$
Expand the the products from the norm and subtract $y'y$
$$wX'Xw - w'Xy - y'X'w = (w - \hat w)X' X (w - \hat w) + \hat w X'X\hat w - \hat w'Xy - y'X'\hat w$$
Terms of dimension $1 \times 1$, are trivially symmetric $w'Xy = y'Xw$, and $\hat w' X y = y' X \hat w$
$$wX'Xw - 2w'Xy = (w - \hat w)X' X (w - \hat w) + \hat w X'X\hat w - 2 \hat w'Xy$$
Then we have to recognize (it may take hours) that $b = Xy$, $x=w$ and $M = (X'X)$, and consequently $\hat w = M^{-1} b$
And thus
$$ x M x - 2(x - M^1b)b = (x - M^{-1}b)M(x - M^{-1}b) + b'M^{-1}MM^{-1}b - 2 b' M^{-1} b $$
Simplifying the product $M^{-1}MM^{-1}$, then grouping the terms $b'M^{-1} b$,
originally $\hat{w} X' X \hat w - 2\hat w' X y$ we cast to the hinted formula
$$ x M x - 2(x - M^1b)b = (x - M^{-1}b)M(x - M^{-1}b) - b'M^{-1}MM^{-1}b$$