Multivariable Calculus and Gradients

Calculus in Many Dimensions

Real problems depend on many variables — a machine-learning model's error depends on thousands of parameters. Multivariable calculus extends derivatives to functions like $f(x, y)$ of several inputs, and its central tool, the gradient, is the engine behind modern machine learning.

A partial derivative measures the rate of change with respect to one variable while holding the others fixed. For $f(x, y)$, the partial $\frac{\partial f}{\partial x}$ is the ordinary derivative treating $y$ as a constant, and $\frac{\partial f}{\partial y}$ treats $x$ as constant. Each partial is a slope along one axis.

The gradient $\nabla f$ collects all the partials into a vector: $\nabla f = \left(\frac{\partial f}{\partial x}, \frac{\partial f}{\partial y}\right)$. Its meaning is geometric and powerful: the gradient points in the direction of steepest increase of $f$, and its magnitude is how steep that increase is. At a maximum or minimum, the gradient is the zero vector — the multivariable analogue of $f'(x) = 0$.

This drives gradient descent, the workhorse optimisation algorithm: to minimise a function, repeatedly step in the direction opposite the gradient (steepest descent), by a small learning-rate amount, until the gradient is near zero. Training a neural network is gradient descent on a loss function of millions of parameters; the gradients themselves are computed by the chain rule (backpropagation). Everything in this unit — derivatives, the chain rule, optimisation — culminates here in the mathematics that powers modern AI.

Common pitfall: taking a partial derivative without holding the other variables constant, and mistaking the direction of gradient descent. When differentiating $\frac{\partial f}{\partial x}$, every other variable is treated as a constant, not as another thing to differentiate. And gradient descent moves opposite the gradient (the gradient points uphill, toward steepest increase) — stepping along the gradient would maximise, not minimise, the function.

A contour map of a bowl with the accent gradient arrow at a point pointing uphill (steepest increase) and a descent step arrow pointing the opposite way toward the central minimum.

$$\nabla f = \left(\frac{\partial f}{\partial x}, \frac{\partial f}{\partial y}\right)$$