Multivariable Calculus and Gradients

Multivariable Calculus

Real problems depend on many variables; a machine-learning model's error depends on thousands of parameters. Multivariable calculus extends derivatives to functions of several inputs, and its central tool, the gradient, is the engine behind modern machine learning.

A partial derivative measures the rate of change with respect to one variable while holding the others fixed. For $f(x, y)$, the partial $\frac{\partial f}{\partial x}$ is the ordinary derivative treating $y$ as a constant, and $\frac{\partial f}{\partial y}$ treats $x$ as constant. Each partial is a slope along one axis.

The gradient $\nabla f$ collects all partials into a vector: $\nabla f = (\frac{\partial f}{\partial x}, \frac{\partial f}{\partial y})$. It points in the direction of steepest increase of $f$, and its magnitude is how steep that increase is. At a maximum or minimum, the gradient is the zero vector.

The Gradient & Descent

This drives gradient descent, the workhorse optimisation algorithm: to minimise a function, repeatedly step in the direction opposite the gradient by a learning-rate amount until the gradient is near zero.

Pitfall: Differentiating with respect to the wrong variable, or forgetting that gradient descent moves opposite the gradient vector because the gradient points uphill.