| Measurement | What it is | Source |
|---|---|---|
| Intended demand | A claim about what the question asks for | Tags you apply |
| Observed difficulty | Evidence about what actually happened | Interview history |
| Comprehension Load | How much English is in the way | Automatic scoring |
The interesting reading is where they disagree. A question tagged
on-level that students consistently stall on has hidden demand. One with low
Comprehension Load and poor grades is probably unclear rather than hard. Averaging the
three would destroy exactly the signal worth having.
The tag vocabulary
Fixed, not free text. Free text fragments immediately — transfer,
Transfer, transfer-question — and the grouping value evaporates.
Grouping is the whole point: per-question data takes a year to become meaningful, but
transfer questions as a class accumulate evidence fast, so a brand-new
question can inherit an expectation from its tags before it has ever been asked.
| Group | Asks | Tags |
|---|---|---|
| Demand | What cognitive move the question asks for | definitional, trace, mechanism, comparison, prediction, transfer, counterexample, misconception, evaluation, design, debug |
| Calibration | Where it sits relative to the course | prerequisite, foundational, on-level, stretch |
| Audience | Who it is pitched at | underclassman, upperclassman, graduate |
| Role | What it is for within a set | warm-up, diagnostic, differentiator, recovery |
| Format | The channel it works through | code-reading, code-writing, diagram, verbal-only |
Sorting uses a coarse ordinal — foundational → on-level →
stretch — derived from the calibration tag but stored separately, because deriving
an order from a tag set at query time is guesswork. prerequisite deliberately has no
level: earlier material is not a point on this course's scale.
Follow-ups are peers, not children
A follow-up is often a good related question rather than a probe, and it carries its own
demand. So it is tagged with the same vocabulary and gets its own calibration and observations.
Both are addressed by an (item_type, item_id) pair rather than duplicating five
tables.
Calibration is course-relative
stretch in CS 101 and stretch in CS 452 are different claims.
Calibration is set once on the item as a base value, with a per-course override stored only where
a course disagrees.
The UI shows inherited and explicit values differently — absence of an override means agreement, not “never considered”, and without that distinction there is no way to tell whether you actually looked.
Observed difficulty
Drawn from what happened in interviews. A recorded attempt grades a competency, not a question, so it cannot be the source; per-question performance lives in the session's history as the Q/P/A/M/B/I bucket alongside the time spent.
| Bucket | Rank |
|---|---|
M Mastery | 4 |
A Approaching | 3 |
P Proficient | 2 |
B Basic | 1 |
I Inconclusive | excluded |
Q Quiz Level | excluded |
A outranks P because approaching mastery is the stronger
performance. Q is excluded because it grades a whole interview against a quiz score
rather than an individual question, and counting it would mix two different measurements.
Scoped to the course, never the item alone
A question graded in AI 420 and in CS 255 produces incomparable results, for three reasons that push in different directions and can silently cancel: the population is stronger, the standard is higher, and the question may be an assessment in one course and a warm-up in the other. A single averaged number across both is quietly wrong.
Scoping is to the course rather than the class, so evidence accumulates across every term a question is taught. A per-semester sample never reaches usable size.
A wrong difficulty score is worse than none, because it would be acted on. Drift direction needs ten — a trend drawn through six points is a line through noise.
Exclusions are counted rather than silently dropped. A question shown many times and never gradeable is a real finding about the question, and hiding it inside a denominator would lose it.
The item metrics window
Everything known about one question or follow-up, in one place: its tags, its observed difficulty per course, and its Comprehension Load. Pairings are stated as questions rather than correlation coefficients — with a bank this size a coefficient would be noise with a decimal point, and a number invites belief the sample cannot support.