Full gradient versus stochastic gradient

Problem

For losses $\ell_1(w)=(w-1)^2$ and $\ell_2(w)=(w+1)^2$ , compute the full empirical gradient at $w=2$ and each per-example gradient. Explain why an unbiased uniformly sampled stochastic gradient can still have variance.

Reveal answer or reference solution

The per-example gradients are $2$ and $6$ . The mean, or full, gradient is $4$ . A uniformly sampled per-example gradient has expectation $4$ but takes values $2$ or $6$ , so it has nonzero variance.

Local history

Loading attempts saved in this browser…

Use with your agent

Share this URL and your attempt. Ask the agent to start with a clarifying question or the smallest useful hint.

Tutor me on https://mlprep.iwase.dev/machine-learning/gradient-descent/original-ml-sgd/. If window.mlPrepAgent is available, read attempts for item original-ml-sgd before tutoring. Inspect my attempt, keep the item ID, and do not reveal the full answer first. After a real attempt, append its record and read it back.

Appears in

Gradient Descent Practice