Why an honest result comes with a margin of error

Every test is a little off

If you took the same kind of test on five different days, with different questions, you wouldn't get the same number five times. One morning you slept badly; on another, a guess paid off; on a third, your phone rang in the middle of a hard question. Psychometrics treats every result as the sum of two parts: the value you would get without those fluctuations, and an error no one can eliminate.

This holds for any instrument, including the full batteries administered by psychologists. What changes from one test to another is the size of the error.

The standard error of measurement

The size of the error has a name: the standard error of measurement. It tells you how much a person's results typically vary around their true value, and it depends on how consistent the test is. On a scale with a standard deviation of 15, a test with a reliability of 0.90 has a standard error of about 5 points. Full batteries usually report reliability above 0.95 for the total score, which gives a standard error of about 3 points or less.

In Item Response Theory, which IQNorth uses, the error is not the same across the whole scale. It is smaller where there are many questions with a difficulty close to yours, and larger at the ends, where fewer questions can separate one score from the next. With the 37 questions that count, the standard error is around 5 to 6 points in the middle of the scale.

The 95% interval

To build the range, you add and subtract roughly twice the standard error from the result. With an error of 5 to 6 points, that gives a range of about 10 points on each side. The name 95% confidence describes the procedure: ranges built this way contain the true value in about 19 out of 20 cases.

How to read “101 to 119”

Take the example above, an estimated IQ of 110 with a range of 101 to 119.

• 110 is the best single estimate, the most likely point.
• A true value of 104 or 116 is perfectly consistent with this result.
• Two people who score 108 and 112 on the same test can't be considered different: their ranges overlap almost entirely.
• The range is not a guarantee. In about 1 case in 20, the true value falls outside it.

Why a short online test has more error

Three things weigh in. The first is length: fewer questions mean less information, and each single right or wrong answer moves the result more. The second is the setting. No one supervises the test, so noise, notifications and fatigue get into the measurement. The third is the norm: IQNorth's is still provisional, built from published ICAR parameters with corrections and from the age curve in the literature (Salthouse, 2009), and it will be recalculated with real responses.

None of these limitations disappears behind a nice-looking number. They show up in the width of the range, which is the right place for them.

Part of the error, though, is up to you. Taking the test somewhere quiet, without interruptions, with your phone on silent and no rush, reduces the noise the test has no way to correct. Time doesn't count toward your score, and reading each question calmly is the best way to make the number reflect your reasoning.

Why a number without a margin misleads

A bare number suggests a precision the test doesn't have. It invites people to compare one another over differences of two or three points, which sit inside the error, and to read a normal fluctuation between two attempts as progress.

That's why the IQNorth retest only flags a change when the difference exceeds the margin, after accounting for the practice effect. In most cases the result comes out stable, and that is what you should expect.

The range doesn't weaken the result. It tells you how much of it can be stated.

See also: Why the result comes as a range.

Read next: How to tell if an online IQ test is reliable