Practice question · Multiple choice
Gradient descent takes a step proportional to the derivative. Why does a learning rate that is too large make the loss increase?
Hints
- The derivative is local information. Ask how far it remains accurate.
- Picture a narrow valley. What does a huge step downhill do?
Show the answer
D. Because the derivative describes the slope only near the current point
Why
A derivative is a statement about an infinitesimal neighbourhood, and a large step leaves the region it describes, in a narrow valley you cross the bottom and climb higher than you started. That is why learning rate is the most consequential hyperparameter, and why schedules and adaptive methods exist.
Practise Derivatives and Rates of Change
The app has 6 more questions on this lesson, and keeps your place in the course. Computer Science I is free to start.