The Memory Hierarchy
Fast storage is small and expensive; large storage is slow and cheap. Computers bridge this gap using a memory hierarchy of stacked layers.
| Level | Speed | Size | Cost |
|---|---|---|---|
| Registers | Fastest | Bytes | Highest |
| Cache | Very fast | KB-MB | High |
| RAM | Fast | GB | Medium |
| SSD / Disk | Slow | TB | Low |
A cache is a small, fast memory sitting between the CPU and RAM. It holds copies of recently used data, not unique data.
Common pitfall: A cache never stores separate data from main memory. It only keeps fast copies of what lives in RAM to speed up access.
Locality and Hits
When the CPU needs data, it checks the cache first. A cache hit is fast, while a cache miss forces a slow fetch from main memory.
Caches succeed because programs show locality of reference:
| Locality Type | Meaning | CPU Action |
|---|---|---|
| Temporal | Used recently? | Use again soon |
| Spatial | Address accessed? | Fetch nearby block |
The hit rate is the fraction of accesses found in the cache. Because misses are so slow, even small drops in hit rate hurt performance.