Inner Product Spaces and Orthogonality

Abstract Geometry and Orthogonality

The dot product of Unit 3 gave $\mathbb{R}^n$ its geometry. An inner product $\langle u, v \rangle$ generalizes this to any vector space. It must be symmetric, linear, and positive definite ($\langle v, v \rangle > 0$ for $v \neq \mathbf{0}$). From it come the norm $\|v\| = \sqrt{\langle v, v \rangle}$ (length) and the angle via $\cos\theta = \frac{\langle u, v \rangle}{\|u\|\|v\|}$.

Two vectors are orthogonal when $\langle u, v \rangle = 0$. An orthonormal basis is orthogonal and every vector has length 1. In such a basis, coordinates are trivial to compute: the $i$-th coordinate of $v$ is $\langle v, e_i \rangle$.

Projections and Gram-Schmidt

Orthogonal projection finds the closest point in a subspace $W$ to a given vector. The error $v - \operatorname{proj}_W v$ is orthogonal to all of $W$. This drives least-squares approximation: when $A\mathbf{x} = \mathbf{b}$ has no exact solution, projecting $\mathbf{b}$ onto the column space gives the best approximate fit.

The Gram–Schmidt process systematically turns any basis into an orthonormal one by subtracting components along chosen vectors.

Common pitfall: Assuming an orthogonal projection is just truncating extra coordinates. Projection finds the nearest vector in $W$ via a geometric construction, and coordinates are trivial only in an orthonormal basis, not a general one.