Assessments at Scale

Machine-scored questions can test reasoning

Prof. Madhavan Mukund starts by asking what assessment is for. It can check whether students understand the prerequisites, show teachers where instruction has failed, certify achievement, assign grades and, in some settings, rank candidates. Open responses reveal reasoning. They also take time to read and are hard to mark consistently across a very large group. Carefully designed machine-scored questions can test understanding at scale.

The examples come from algorithms and data structures. A multiple-choice question might test whether a student understands asymptotic complexity as a bound, can connect notation with running time, or can distinguish insertion sort from selection sort. Other questions compare Prim's and Kruskal's algorithms or reverse the usual reasoning around a heap operation. A good item puts the concept into a situation where a common misconception leads to a plausible wrong answer. This makes writing the question demanding: a small substitution can change the correct answer, so the setter must verify the stem, every option and the reasoning behind them.

Machine-scored assessment offers several formats. Multiple-correct questions can reduce the value of guessing and allow partial credit when a student chooses some correct options and avoids every incorrect option. A fixed-format answer, such as a number or sequence, can also be generated by the student and scored automatically. Computing Olympiad problems use increasingly difficult test cases to indicate whether an algorithm is efficient. CMI questions leave gaps in a proof outline. Staged questions can move from a recurrence to evaluation order and complexity. Programming assignments can be judged online.

Useful questions may come from textbooks, lectures, daily life, public problem sets or culturally adapted versions of existing items. Whatever the source, students must have had a fair chance to learn the material. Writing these questions tests the teacher too: exposing a misconception requires a firm grasp of the idea, its edge cases and its plausible alternatives. Design assessment alongside the teaching.