ML Prep

Matrix derivative with a numeric check

Problem

Let f(w)=12∥Xw−y∥22f(w)=\tfrac12\|Xw-y\|_2^2, where

X=[1011],y=[12]X=\begin{bmatrix}1&0\\1&1\end{bmatrix},\quad y=\begin{bmatrix}1\\2\end{bmatrix}

Derive the gradient and Hessian for general ww, then evaluate the gradient at w=(0,1)Tw=(0,1)^{\mathsf T}.

Reveal answer or reference solution

Expanding the differential, or applying the vector chain rule, gives

∇f(w)=XT(Xw−y),∇2f(w)=XTX\nabla f(w)=X^{\mathsf T}(Xw-y),\qquad \nabla^2 f(w)=X^{\mathsf T}X

Here

XTX=[2111]X^{\mathsf T}X=\begin{bmatrix}2&1\\1&1\end{bmatrix}

At w=(0,1)Tw=(0,1)^{\mathsf T}, Xw−y=(−1,−1)TXw-y=(-1,-1)^{\mathsf T}, so

∇f(w)=XT(−1,−1)T=(−2,−1)T\nabla f(w)=X^{\mathsf T}(-1,-1)^{\mathsf T}=(-2,-1)^{\mathsf T}
Outcome

Previous attempts

Loading attempts saved in this browser…

    Review Screening Results