Illusion of Validity
The illusion of validity is confidence produced by a selection ritual’s intensity and structure rather than by evidence that it predicts performance. A test can feel highly revealing while measuring abilities unlike those required in the real role.
Two punishing weeks in an Israeli military boot camp seemed capable of revealing who would become a good officer. Yet performance in the camp had no correlation with later officer performance.
E1Difficulty masquerades as prediction
Vivid, demanding evaluations generate observations that feel diagnostic: who stays composed, takes charge, or impresses assessors under pressure. Their formality and intensity make those judgments feel valid. But predictive value depends on something less theatrical—whether the test recreates the abilities and conditions that determine success afterward. When selection and real work demand different things, confidence in the ranking can remain high even as its signal approaches zero. This is the selection-stage cousin of Performance-Potential Mismatch: demonstrated performance transfers only when the next environment rewards the same capabilities.
E1The problem is mismatch, not rigor
This case does not show that intense selection is always useless. It shows that intensity alone establishes nothing: a demanding test may predict later performance when its tasks genuinely match the role. Nor does one military case establish how every selection system performs; each system needs its own outcome evidence.
E1Make the ritual predict something
For any test you control, name the later outcome it is supposed to predict and compare scores with actual performance in that role. If the relationship is absent, redesign the test around the work instead of defending it because it is difficult.
E1Episodes that teach this
-
How The World Chooses To Be Incompetent - The Peter Principle - FutureIQ
10,594 views
A two-week intense Israeli military boot camp showed “no correlation” with later officer performance.