Introduction to Probability

Quantifying Uncertainty

Probability measures how likely an event is, on a scale from 0 (impossible) to 1 (certain). It is the mathematics of uncertainty, underlying statistics, physics, finance, and machine learning.

The setup:

• The sample space $S$ is the set of all possible outcomes of an experiment (for a die, $\{1, 2, 3, 4, 5, 6\}$).
• An event is a subset of the sample space (e.g. "even," $\{2, 4, 6\}$).
• The probability $P(E)$ is a number in $[0, 1]$ assigned to each event.

For equally likely outcomes, probability is a ratio of counts: $$P(E) = \frac{\text{number of favorable outcomes}}{\text{total number of outcomes}}.$$ So $P(\text{even}) = \frac{3}{6} = \frac{1}{2}$ for a fair die. This connects probability directly to the counting techniques of combinatorics.

Probability obeys three axioms (Kolmogorov's):

1. $P(E) \ge 0$ for every event.
2. $P(S) = 1$ — some outcome certainly occurs.
3. For mutually exclusive events (disjoint, cannot both happen), probabilities add: $P(A \cup B) = P(A) + P(B)$.

From these follow all the rules, including the complement rule: $P(\text{not } E) = 1 - P(E)$ — often the easiest route ("at least one" problems are usually solved via "none"). For events that can overlap, the general addition rule subtracts the double-counted intersection: $P(A \cup B) = P(A) + P(B) - P(A \cap B)$ — the same inclusion-exclusion seen with sets.

Common pitfall: adding probabilities as $P(A) + P(B)$ when the events can overlap. Simple addition is valid only for mutually exclusive events. If $A$ and $B$ can both occur, you must subtract the overlap: $P(A \cup B) = P(A) + P(B) - P(A \cap B)$, or you double-count the shared outcomes and can even exceed 1.