Inner Product Spaces and Orthogonality

Length and Angle in Abstract Spaces

The dot product of Unit 3 gave $\mathbb{R}^n$ its geometry — lengths and angles. An inner product $\langle u, v \rangle$ generalizes this to any vector space, including spaces of functions. It must be symmetric, linear in each argument, and positive definite ($\langle v, v \rangle > 0$ for $v \neq \mathbf{0}$). From it come the norm $\|v\| = \sqrt{\langle v, v \rangle}$ (length) and the angle via $\cos\theta = \frac{\langle u, v \rangle}{\|u\|\,\|v\|}$. This closes the unit by returning geometry to abstract spaces.

Two vectors are orthogonal when $\langle u, v \rangle = 0$ — the generalized perpendicularity. A set of mutually orthogonal nonzero vectors is automatically linearly independent, so orthogonal sets make especially convenient bases. An orthonormal basis is orthogonal and each vector has length 1; in such a basis, coordinates are trivial to compute — the $i$-th coordinate of $v$ is just $\langle v, e_i \rangle$ (no system to solve).

Orthogonal projection finds the closest point in a subspace to a given vector. The projection of $v$ onto a subspace $W$ is the vector in $W$ nearest to $v$, and the error $v - \operatorname{proj}_W v$ is orthogonal to all of $W$. This is the geometric heart of least-squares approximation: when a system $A\mathbf{x} = \mathbf{b}$ has no exact solution, the least-squares solution projects $\mathbf{b}$ onto the column space, giving the best approximate fit — the foundation of data fitting and regression.

The Gram–Schmidt process turns any basis into an orthonormal one, systematically subtracting off components along the vectors already chosen. Orthonormal bases and projections make computation clean and are central to Fourier analysis, least squares, and much of applied mathematics — a fitting close to the linear-algebra unit.

Common pitfall: assuming an orthogonal projection is just "drop the extra coordinates," or that any set of vectors gives easy coordinates. Projection onto a subspace $W$ finds the nearest vector in $W$, with the residual $v - \operatorname{proj}_W v$ orthogonal to $W$ — it is a specific geometric construction, not mere truncation. And coordinates are trivial ($\langle v, e_i \rangle$) only in an orthonormal basis; in a general basis you must solve a linear system.