Item Response Theory without the formulas
The same family of models that scores the ENEM, Brazil's national high school exam, calculates IQNorth's estimate. The idea fits in a few pictures, with no equations at all.
In this article
The problem with just counting correct answers
The simplest way to score an exam is to count correct answers. It treats every question as equal, and they aren't. Getting right a question that almost everyone gets right says little; getting right one that almost everyone gets wrong says a lot.
Two test takers with the same number of correct answers may have taken very different tests. One got the easy and medium questions right and missed the hard ones. The other missed some easy ones and got scattered hard ones right. The count calls it a tie. Item Response Theory, or IRT, breaks that tie by looking at which questions were answered correctly.
Every question has a curve
In IRT, each question gets its own curve, the item characteristic curve. The horizontal axis shows the test taker's ability; the vertical axis, the chance of getting it right. For someone with little ability, the chance is low; it rises in one region of the scale and approaches the maximum for someone with a lot.
The curve is S-shaped. Where it sits and how sharply it rises vary from one question to another, and those differences are what the model uses.
Think of two matrix questions. In one, the rule shows up in the first row and almost every adult spots it; the curve rises early, far to the left. In the other, two rules act at the same time and only reveal themselves together; the curve stays flat until the upper part of the scale and only then rises. A correct answer on the second one says much more about the person who answered it.
The three parameters
IQNorth uses the three-parameter model, the same kind used in large-scale testing. Each question is described by three numbers.
- Difficulty: the point on the scale where the curve rises. A hard question only starts being answered correctly with any frequency by people with more ability.
- Discrimination: how steep the curve is. A question that discriminates well sharply separates people slightly below its difficulty point from those slightly above it.
- Guessing: the floor of the curve. On a multiple-choice question, even someone who doesn't know the way gets it right now and then, and the model accounts for that chance.
Each question's parameters come from data: they are estimated from the responses of many people before the question is used to score anyone.
Why getting the hard ones right is worth more
The estimate of your ability is the point on the scale that best explains your set of right and wrong answers. The model tries out values and keeps the one at which your response pattern would be most likely.
A correct answer on an easy question is expected from almost everyone and barely moves that estimate. A correct answer on a hard, highly discriminating question is only likely for someone high on the scale, and it pushes the estimate up. That's why the result is not a stretched percentage of correct answers.
The same reasoning explains why the margin of error varies. Where the test has many questions with a difficulty close to yours, each answer helps locate you more reliably. Far from them, at the ends of the scale, there is less information, and the range around the number widens. IRT calculates this error point by point, instead of using a single value for everyone (Embretson and Reise, 2000).
In IRT, what counts is which questions you got right, not just how many.
Why guessing can't produce a high number
Someone who guesses gets a few hard questions right by luck, at the rate the guessing parameter already predicts. These isolated hits, surrounded by misses on easier questions, form a pattern the model attributes to chance. That is exactly what the third parameter is for: it reduces the weight of a correct answer in a region where luck explains a good part of it.
The ENEM has used the same reasoning since 2009, when INEP adopted three-parameter IRT. That's why, on the ENEM, two test takers with the same number of correct answers can get different scores: what counts is the consistency between what they got right and what they got wrong.
When the pattern doesn't make sense
Sometimes a set of answers is unlikely at any point on the scale: many misses on easy questions and many hits on hard ones. This can come from distraction, impulsive answers or someone helping with part of the test. In any case, the calculated number deserves less trust.
On IQNorth, a very unusual pattern, or answers too fast to have been read, gets the result flagged as unreliable. Retaking it is free.
See also: The method.
Sources
- Birnbaum, A. (1968). Some latent trait models and their use in inferring an examinee's ability. In F. M. Lord & M. R. Novick, Statistical Theories of Mental Test Scores. Reading, MA: Addison-Wesley.
- Lord, F. M. (1980). Applications of Item Response Theory to Practical Testing Problems. Hillsdale, NJ: Erlbaum.
- Hambleton, R. K., Swaminathan, H., & Rogers, H. J. (1991). Fundamentals of Item Response Theory. Newbury Park, CA: Sage.
- Embretson, S. E., & Reise, S. P. (2000). Item Response Theory for Psychologists. Mahwah, NJ: Erlbaum.
