Series: curiosity · 2026-10-18
A rare thing in its honesty appeared on Habr: a retrospective by an engineer who dissected his own failure. He was building a dual-stream model to determine the stage of diabetic retinopathy: the input was a fundus image and the patient’s clinical indicators. On the held-out test set — QWK 0.834, an excellent metric, everyone happy. Then the author started looking into what, exactly, made the model accurate, and discovered something unpleasant: the RetinaMNIST dataset contains no clinical test results by design — he had generated them himself, using a formula that included the true stage of the disease. The clue had been sewn directly into the features.
The control experiments laid it all out. Logistic regression, which saw only the “clinical” data and not a single image pixel, got QWK 0.816 — nearly catching the dual-stream network (0.834). And an honest network trained on images alone scored 0.713. In other words, the neural network was not learning to diagnose — it was learning to read the note with the answer, neatly slipped into the feature table. That note has a name: data leakage. What follows is a journey from a purple marker in the corner of a photograph to hundreds of scientific papers that will probably have to be read with this thought in mind.
The most vivid leakage case I know of was described in JAMA Dermatology (Winkler et al., 2019). The team took an off-the-shelf medical CNN for melanoma diagnosis and ran it on archival photographs of skin lesions. On images without markings, the model performed beautifully: sensitivity 95.7%, specificity 84.1%.
The catch lies in what clinical archives look like. Surgeons circle suspicious lesions with a purple marker — before a biopsy or removal. In the frame, it is just a tiny purple dot, but statistically it is inextricably correlated with the diagnosis: after all, they mostly mark what the doctor has already deemed suspicious. The model learned the rule “purple = cancer,” with devastating consequences: on marked images, specificity collapsed to 45.8%, while the average “confidence in melanoma” for clearly benign nevi jumped from 0.16 to 0.54. Heat maps (vanilla gradient backpropagation) showed it plainly: the model’s attention peaks landed precisely on the blue pixels of the markings.
An elegant detail: if you crop out the background and leave only the lesion itself, performance shoots up to AUC 0.993 — higher than the original model’s 0.969. All along, the model had been keeping the background in mind, not the skin. One purple stroke cost it thirty-eight percentage points of specificity.
The purple marker is crude, visible leakage. Zech et al. (PLOS Medicine, 2018) showed how it can work more subtly. The authors assembled 158 323 chest X-rays from three institutions: NIH (112 120 images), Mount Sinai Hospital (42 396), and Indiana University (3 807) — and trained CNNs to detect pneumonia. Inside their “own” hospitals, everything was excellent: the combined MSH+NIH model achieved AUC 0.931. But on the external Indiana dataset, that same neural beast produced 0.815 — noticeably worse.
The answer is ugly. The three hospitals differ radically in pneumonia prevalence: at Mount Sinai, 34.2% of patients are ill; at NIH and Indiana, 1.2% and 1.0%. As a result, even the trivial task of “sort the images by hospital” gives AUC 0.861 — better than many respected computer-aided diagnostic systems. And the CNN eagerly learns this signal: a separate network identifies an image’s home institution with 99.95% accuracy for NIH and 99.98% for MSH. Equipment, protocols, patient populations — all of it leaves a hospital “passport” in the pixels, and the model faithfully migrates toward the most reliable correlate of the target it can find. The diagnosis has nothing to do with it: the model learned the passport, not pneumonia.
Sara Beery and co-authors (ECCV 2018, “Recognition in Terra Incognita”) ran the same experiment in the wild: 20 camera traps across three U.S. nature reserves, hundreds of thousands of wildlife images. The task was to learn to recognize species and transfer that skill to a new camera never seen before. The result is sobering: on familiar locations, state-of-the-art models are wonderful; on new ones, they fail. A camera in a different thicket, with a different background and lighting — and it is already a different universe. Model learned the scenery, not the species: the same mechanism as with the hospital passport, only now the “passport” is the shrubs and shadows of one particular clearing.
There is another side to the coin: the split can be flawless, and the test can still be blind. Luke Oakden-Rayner and co-authors called this hidden stratification: the overall metric is high, but subgroups lurk within the data that nobody thought to slice it by — and the model fails on them consistently (in their work, relative drops of more than 20% on clinically important subtypes). The subtype “rare aggressive cancer” sits cozily inside the broad class “cancer” and ruins the lives of those who need to catch precisely that kind.
Then there is the quality of the datasets themselves. In ChestXray-14 (112 120 images), the labels were not assigned by a physician: they were extracted through automated data mining of report texts. Oakden-Rayner’s team rechecked a representative sample — 50 cases per class, 700 in all — and the labels’ positive predictive value turned out, for the most part, to be 10–30 percentage points lower than claimed. A model trained on that honestly learns — just not what the column heading says.
Individual cases could be written off as flukes, but Princeton statisticians Sayash Kapoor and Arvind Narayanan assembled the broader picture in Patterns (2023): across 22 review papers, they documented leakage in at least 294 articles across at least 17 scientific fields — from medicine to political science. They also proposed a taxonomy of 8 types of leakage: from the classic “preprocessing before the split” to subtleties like feature selection on the full dataset.
Their most high-profile dissected example was the prediction of civil wars in political science. Of 124 publications in the field, only 12 made their code and data public. Of those twelve, four contained errors leading to leakage — exactly the ones most loudly proclaiming the superiority of complex ML over the classics. Once the leaks were fixed, ML’s advantage over logistic regression (a model almost a century old) vanished; in the most striking pair, the AUC gap shrank from 0.14 to 0.01. The cherry on top: 9 of the 12 papers had no uncertainty estimate at all — the claimed “AUC 0.85” in one of them had a 95% CI of 0.66–0.95, meaning it was not statistically distinguishable from the baseline. An unpleasant stroke in the portrait of the field: irreproducible papers are cited more often than reproducible ones. Error is rewarded with attention.
While scientists were breaking lances, industry showed how to climb the rankings with no data at all. In his 2015 post “Competing in a data science contest without reading the data,” Moritz Hardt pulled off a trick called wacky boosting in a real Kaggle tournament, the Heritage Health Prize, with a $3M prize pool and a two-year duration. The recipe: generate random binary prediction vectors, keep the ones that get a little lucky on the public leaderboard, and take a majority vote. After ~700 submissions, the “method” improved its position from ~146th place to ~6th, without ever looking at the data. The leaderboard score is rounded — and over tens of thousands of repeated measurements, this is simply a random-number machine with deliberate selection.
Peter Blum and Hardt later formalized the diagnosis theoretically (The Ladder, ICML 2015; Dwork et al., The Reusable Holdout, Science 2015): a holdout set is honestly valid for one single query. A leaderboard is thousands of implicit queries against it, and sooner or later luck disguises itself as quality. Kaggle’s standard defense is to recalculate the final ranking on a closed private split, but the leakage story does not end there: in the WSDM-KKBox Churn Challenge 2017 (predicting subscriber churn for a music service), a discrepancy surfaced between the official test labels and the data by which they were actually calculated; the author of the thread about it, Dan Ofer, described the organizers’ reaction with the word “embarrassment” — and the labels were never fixed. InfiniteWing, who finished 17th, noted in a public solution write-up that self-generating labels gave him ~0.0015 improvement in the metric. Leakage is not always in a participant’s code — sometimes it is in the organizer’s labels.
The remedy is known, but it is discipline, not a spray:
I like this story because leakage is not about stupidity or cheating. The machine does exactly what it was asked to do: minimize error on a sample where the answer lies nearby — like a cheat sheet under the pillow during an exam. It does not read the cheat sheet consciously; the cheat sheet was simply lying in plain sight, and the model that spotted it automatically won. A purple marker, a hospital passport, a timestamp from the future — everywhere, the mechanism is the same: the data-collection procedure created a channel through which the answer flows into the features.
Human analysis gets stuck on the comforting narrative that “the model learned pathology.” The model learned our procedure — and it is indistinguishable from something smart until it is tested on data that, by construction, could not have leaked into training. Healthy skepticism here is hygiene, not paranoia: any unexpectedly beautiful number must survive on an external block before it is carried into a journal or into production.
Sources: Habr retrospective on retinopathy (habr.com/ru/articles/1095572, QWK 0.834→0.713); Winkler et al., JAMA Dermatology 155(10), 2019 (specificity 84.1%→45.8%, nevus score 0.16→0.54, AUC 0.969/0.922/0.993); Zech et al., PLOS Medicine 15(11), 2018 (158 323 images, NIH/MSH/IU, prevalence 34.2/1.2/1.0%, hospital-ID 99.95/99.98%, AUC 0.931→0.815); Beery, van Horn, Perona, “Recognition in Terra Incognita,” ECCV 2018; Oakden-Rayner et al., Hidden Stratification (ACM CHIL 2020), Exploring Large-scale Public Medical Image Datasets (Academic Radiology, 2019; sample of 700 cases, PPV lower by 10–30 p.p.); Kapoor & Narayanan, “Leakage and the Reproducibility Crisis in ML-based Science,” Patterns 4, 2023 (294 articles, 17 fields, 22 reviews, 124→12→4, AUC gap 0.14→0.01); Moritz Hardt, “Competing in a data science contest without reading the data,” blog.mrtz.org, 2015 (Heritage Health Prize, 146→6 in ~700 submissions); Blum & Hardt, The Ladder (ICML 2015); Dwork et al., The Reusable Holdout (Science, 2015); InfiniteWing & Dan Ofer, KKBox Churn Challenge discussion, Kaggle 2017; Roberts et al., “Cross-validation strategies for data with temporal, spatial, hierarchical, or phylogenetic structure,” Ecography 40: 913–929, 2017.