Why retaking the test almost always comes out “stable”

The practice effect

Taking the same kind of test twice almost always improves the second score. The first time, part of your effort goes into understanding the format: how the figures are arranged, how much time each question takes, where the trap usually is. The second time, all of that is already figured out, and more attention is left for the problem itself.

The phenomenon is well measured. In 2007, John Hausknecht and colleagues gathered retest studies of cognitive ability tests used in personnel selection and found an average gain of about a quarter of a standard deviation (0.26), smaller when the second administration used an alternate form with different questions. In 2018, Jana Scharfen, Jan Peters and Heinz Holling found a gain of about a third of a standard deviation from the first administration to the second, with smaller gains after that, leveling off around the third.

On the IQ scale, where one standard deviation is 15 points, a quarter of a standard deviation comes to nearly 4 points. That sounds small, but it is enough to look like real improvement to anyone who does not know about the effect.

The error of each test adds up

Every result has measurement error. On IQNorth, with 37 scored questions, the standard error is around 5 to 6 points in the middle of the scale, which is why the number comes with a 95% confidence range of about 10 points on either side.

When you compare two tests, both errors enter the calculation. The error of the difference is larger than that of either test alone: it is the square root of the sum of the squares of the two errors. With two errors of 5 points, the error of the difference is over 7. A difference of 6 or 8 points between two administrations, then, fits entirely within the noise.

There is also regression to the mean. Someone whose first result was far from 100 tends to land a little closer to 100 the second time, because part of the initial deviation was that day's luck, good or bad. This holds on both sides of the curve.

A difference between two tests only means something when it clearly exceeds the combined error of both plus the practice effect.

Parallel form: new questions, same yardstick

Retaking exactly the same questions mixes two effects: practice with the format and memory of the answers. To remove the second, the IQNorth retest uses a parallel form. Every question on the first test has a twin on the second form, with the same rule structure and the same kind of trap, but different content. In the matrices, the shapes and fills change; in the series and logic items, the rule repeats with other numbers and words; in the memory sequences, the digits are new.

The idea is to measure the same ability with the same yardstick. Because both forms are scored with Item Response Theory, each question counts according to its own difficulty, and the results land on the same scale. The equivalence of the two forms is still a hypothesis, which will be checked question by question when the norms are recalculated with real responses.

The retest opens 30 days after the test. The interval does not erase the practice effect, but it lowers the chance that you remember details from the first time.

How the comparison decides

The comparison follows three steps, in this order.

1. It discounts practice. From the difference between the two results, we subtract the gain expected from repetition, based on Hausknecht (2007) and Scharfen (2018). The discount is about 4 points on the first retake and grows slightly on later ones, up to a plateau.
2. It calculates the margin. We combine the errors of the two tests to get the error of the difference, as described above.
3. It decides. We only say your score went up or down if the difference, with practice already removed, exceeds about twice that error. This is the reliable change criterion proposed by Neil Jacobson and Paula Truax in 1991. Below that, the result comes out stable.

The report also warns you when something could distort the comparison: a short interval, use of training in the same format, a first result very far from 100, a result flagged as unreliable, or one near the test's ceiling.

Why almost everyone comes out “stable”

With these rules, the retest is conservative on purpose. In simulations of our own method, a real change of 10 points was detected about a quarter of the time; one of 15 points, about half the time; one of 25 points, more than 9 times out of 10. When there was no real change, the method reported a false increase in fewer than 2 cases out of 100.

So the retest only sees large changes. That is the price of not mistaking practice and noise for change. And large changes in an adult's reasoning over a few months are rare: differences between people are quite stable across the lifespan, as the article on age and reasoning shows.

“Stable” does not mean nothing has changed in your life. It means that, with this measure, the difference cannot be separated from error and practice.

See also: Parallel-form retest.