The Pyramid of Storage
There is a brutal trade-off in memory: fast storage is small and expensive, while large storage is cheap and slow. No single technology is both fast and huge, so computers use a memory hierarchy — a stack of layers, fast-and-small at the top, slow-and-large at the bottom.
| Level | Speed | Size | Cost per byte |
|---|---|---|---|
| Registers | Fastest | Bytes | Highest |
| Cache (L1, L2, L3) | Very fast | KB–MB | High |
| Main memory (RAM) | Fast | GB | Medium |
| Disk / SSD | Slow | TB | Low |
A cache is a small, fast memory that sits between the CPU and main memory and holds copies of recently or frequently used data. When the CPU needs a value, it checks the cache first. A cache hit (the value is there) is fast; a cache miss forces a slow trip to main memory — and the fetched block is then copied into the cache for next time.
Caches work because real programs exhibit locality of reference. Temporal locality: a value used now is likely to be used again soon (so keep it close). Spatial locality: if one address is used, nearby addresses are likely to be used soon (so caches fetch a whole block at once, not one byte). The fraction of accesses that hit the cache is the hit rate, and because a miss is so much slower than a hit, even a small drop in hit rate can badly hurt performance.
Common pitfall: imagining the cache stores different data than main memory. It does not — a cache holds fast copies of data that also lives in main memory. Its value is purely speed: keeping the data the program is likely to touch next in the layer closest to the CPU.