Where Sports Analytics Actually Beats Judgement

Every winter someone declares that analytics has ruined sport, and every spring someone declares that anyone still trusting their eyes is a fossil. Both camps are arguing about the same word as though it named one activity. It does not.

The argument predates sport by fifty years. Paul Meehl's Clinical versus Statistical Prediction (1954) gathered roughly twenty head-to-head comparisons between expert clinicians and simple actuarial formulas, and the formulas won or tied nearly every time. The finding held up under scale: Grove, Zald, Lebow, Snitz and Nelson's meta-analysis of 136 studies found mechanical prediction about 10% more accurate on average than clinical judgement, and rarely worse.

That is the honest baseline. It is a real, replicated advantage. It is also 10%, not a revelation.

Injury forecasting is the clearest failure case. Van Eetvelde and colleagues' systematic review of machine-learning injury prediction (Journal of Experimental Orthopaedics, 2021) found eleven qualifying studies with performance ranging from AUC 0.87 down to 0.52 — indistinguishable from a coin flip. Three of eleven were near chance. Most used small single-team samples with no external validation.

We work the same ground in Does training load predict injury risk?, where the methodological problem turns out to be worse than the sample-size problem.

Meehl's own literature contains the broken-leg case: the actuarial table predicts a man will attend the cinema tonight, the clinician knows he broke his leg this morning. When a human holds decisive information the model has no column for, the human should win — and in Grove's tally, clinical judgement did match or beat the model in a small but non-trivial minority of comparisons.

The second counterargument is the Marcel result read the other way. If a monkey-simple model matches an expensive one, the marginal value of modelling sophistication in this domain is small, and the industry's confidence in its own machinery is not fully earned.

A pre-registered, blinded tournament pitting scouting boards against models on the same draft classes, with success metrics agreed in advance, would settle the draft question properly. None exists. Single-cycle demonstrations that an algorithm "beat the scouts" are not that study. On the injury side, large multi-team datasets with genuine external validation would tell us whether the low AUCs reflect data scarcity or a genuinely unpredictable event.

None of this shows models are superior for single high-stakes, low-sample decisions. It shows they are superior on average, across many cases, where history is abundant. Those are different claims, and conflating them is how a front office talks itself into certainty it has not bought.