Practice question · Multiple choice
A word-frequency program is asked to handle a 50 GB file that will not fit in memory. Which part of the design has to change?
Hints
- Ask which data structure has to hold everything, and which only has to hold what it has seen.
- How large is the set of distinct words compared with the text?
Show the answer
A. The assumption that the whole input is available at once
Why
Distinct words grow far more slowly than text, so the counter fits comfortably while the file does not, process a line at a time and the memory ceiling disappears. Recognising which structure must scale with input and which need not is the whole of the change, and it generalises to most data processing.
Practise Integrated Programming Synthesis
The app has 5 more questions on this lesson, and keeps your place in the course. Computer Science I is free to start.
More questions on Integrated Programming Synthesis
- A word-frequency counter needs to split text, normalise case, count occurrences and report the top results.…
- Order the tasks of building the word-frequency counter so that each one can be tested as soon as it is…
- Sort each part of the word-frequency program by whether it can be written as a pure function or must carry…
- Select every edge case the lesson says the word-frequency counter's tests should cover.