An Alzheimer’s MRI model scored 94.5%. On new data, it fell to 61.6%
For a patient, the important test is whether a model works beyond familiar images. This classifier’s accuracy dropped sharply on an external dataset, and it remains a research system rather than a clinical diagnostic tool.
The short version
- What happened
- Internal accuracy was 94.5% on a balanced 292-slice test set, but fell to 61.6% on a separate balanced 292-image collection without fine-tuning.
- Who it affects
- Patients should not receive a dementia stage from this model; it has not been tested as a clinical diagnostic system.
- What remains unproven
- The OASIS slice collection lacked subject identifiers, preventing a patient-independent split.
Keep in mind
The OASIS slice collection lacked subject identifiers, preventing a patient-independent split.
The internal result came from one fixed balanced split of 292 slices, not repeated patient-level validation.
External validation used 292 images from a Kaggle collection rather than a clinically governed multi-site cohort.
The unit was an image slice, not a patient or complete diagnostic episode.
No prospective workflow, clinician comparison, calibration assessment or patient outcome was tested.
Source analysis · 1 linked source. Primary analysed source date: 13 October 2026.
Read the full report · 6 min The evidence, methods, implications and limitations
The harder test is the one that matters
An AI model designed to classify Alzheimer’s disease stages from brain scans proved much less accurate when tested on images from another collection. For clinicians considering such systems, the study offers a warning: the first headline score may say little about performance beyond the data a model already knows.
The peer-reviewed engineering study reports 94.5% accuracy for a four-stage classifier on its internal test set. Applied without retraining to 292 images from an independent collection, the same model reached 61.6%, with macro-F1 falling to 0.565. The roughly 33-percentage-point gap matters more than the impressive first score: the result did not carry over reliably to the new collection.
The researchers deserve credit for reporting the external test and for stating that the model is not ready for clinical use. The work shows partial transfer, especially for visually distinctive moderate dementia, but performance on the clinically difficult intermediate categories deteriorated sharply. It is a methods study using public image collections, not a prospective diagnostic trial, and it provides no evidence that using the model improves diagnosis, treatment or patient outcomes.[1]
How the model and data were assembled
The system combines an ImageNet-pretrained ConvNeXt-Tiny image backbone with a Kolmogorov–Arnold Network, or KAN, classification head. The head maps a 768-dimensional feature vector to four labels: non-demented, very mild, mild and moderate dementia. Training used mild image transformations, MixUp, CutMix, label smoothing, AdamW optimisation and test-time augmentation.
The publicly distributed OASIS slice collection contained 86,437 axial images: 67,222 labelled non-demented, 13,725 very mild, 5,002 mild and 488 moderate. The researchers capped each class at 1,500 slices, held out balanced validation and test sets before oversampling and replicated only the remaining training images. That avoids the obvious error of duplicating a test image into training and gives each class equal weight in the 292-slice internal test set.[1]
Patient-level leakage remains unresolved
The more important dependence problem could not be eliminated. One three-dimensional MRI scan produces many neighbouring two-dimensional slices, but the public slice collection does not retain per-slice patient identifiers. The study could check that files were not bit-for-bit duplicates across partitions, yet it could not stop slices from the same person appearing in both training and test sets. Similar anatomy across adjacent slices can make internal performance look better than generalisation to a new patient.
Calling the internal protocol leakage-controlled therefore needs a precise qualification: it controls duplication caused by oversampling, not shared-patient leakage. The authors acknowledge this and name a patient-level split as the most important next step. They also used one fixed split rather than repeated hold-outs or cross-validation, so the stability of the 94.5% estimate across alternative partitions is unknown.[1]
The external errors concentrate where staging matters
The independent dataset used the same four labels and contributed 73 images per class. Moderate dementia remained perfectly recalled at 73 of 73 images, and non-demented recall was 79.45%, close to the internal 82.19%. Mild-dementia recall, however, collapsed from 98.63% internally to 10.96% externally. Very-mild recall fell from 97.26% to 56.16%.
Many mild images shifted into the neighbouring very-mild or moderate categories. That pattern is consistent with acquisition differences, contrast, slice selection and label construction moving the decision boundary. It is not enough to say the external set is harder: a model proposed for staging needs trustworthy discrimination at the earlier and intermediate stages, where care decisions are sensitive and visible structural differences are subtle.[1]
The novel head was not the source of a clear gain
In a matched shortened training protocol, the KAN and conventional multilayer-perceptron heads were statistically indistinguishable on accuracy (McNemar p=0.52). The MLP led on several threshold-dependent metrics, while KAN led on ROC-AUC. The KAN head had about nine times as many head parameters, though it increased the full model by only 2.84% and added no material runtime in the reported GPU test.
The paper's ablation instead identifies pretrained features and expanded data use as the principal contributors. Its training-partition resampling also outperformed the tested focal-loss, class-balanced-loss and feature-space SMOTE approaches. Those results are useful engineering evidence, but they weaken any simple claim that the new KAN component produced the headline performance.[1]
What clinicians and health systems should take from it
Clinicians should not use this model to stage a patient. The labels came from public image repositories, the unit of evaluation was an image slice rather than a patient pathway, and the study did not compare predictions with a blinded multi-reader clinical reference. Saliency maps concentrated on plausible brain regions, but no quantitative test established that those maps were faithful explanations or that clinicians could act on them safely.
For health systems, the external decline is a practical demonstration of why local validation must precede deployment. Testing should preserve patient separation, use scans from every intended scanner and population, report calibration and clinically defined errors, and evaluate the entire workflow—including radiologist review, uncertainty handling and the consequences of a wrong stage—not only per-image accuracy.[1]
What would change the assessment
The next evidence should use subject-linked volumetric MRI with patient-level partitions, repeated or cross-validated estimates and multi-site cohorts such as ADNI under an appropriate data-use agreement. A locked model should be evaluated prospectively against clinical diagnoses and independent readers, with age, sex, scanner, comorbidity and disease-spectrum breakdowns. Domain adaptation must be trained separately from the external test it is intended to pass.
Until then, the most important contribution is not a deployable Alzheimer’s classifier. It is a transparent example of how internal success can shrink under distribution shift and how a novel architectural component may add less than disciplined data handling. The external test makes the paper more useful—and the clinical claim much narrower—than the 94.5% headline alone would suggest.[1]
Keep following the evidence
Join The Impact Brief founding list for selected AI reports, their effect on people and the evidence limits. Read a sample first.
Continue the story
Read next
Continue exploring medical imaging: compare the finding and its evidence limits.
A lung-nodule model travelled better—but its labels were not pathology
A Chinese research team improved discrimination and calibration when moving a CT model from a US dataset to a Portuguese cohort. The test covered 618 external nodules, but remained retrospective and partly relied on radiologist ratings.
Read the next reportExplore the full source and evidence record
What this means for people
- Patients should not receive a dementia stage from this model; it has not been tested as a clinical diagnostic system.
- Clinicians need patient-level, multi-site validation and error analysis at early and intermediate stages before considering decision support.
- Health-system buyers should treat external performance and workflow safety as primary evidence, not an internal accuracy headline.
Global context
The study used publicly distributed MRI slices and an independent Kaggle collection rather than a geographically defined clinical cohort. Scanner protocols, populations, diagnostic practice and prevalence differ across sites, so external performance cannot be assumed elsewhere.
What the evidence does not yet show
- The OASIS slice collection lacked subject identifiers, preventing a patient-independent split.
- The internal result came from one fixed balanced split of 292 slices, not repeated patient-level validation.
- External validation used 292 images from a Kaggle collection rather than a clinically governed multi-site cohort.
- The unit was an image slice, not a patient or complete diagnostic episode.
- No prospective workflow, clinician comparison, calibration assessment or patient outcome was tested.
What to watch next
- Subject-level evaluation using volumetric scans and repeated partitions
- Locked multi-site validation on ADNI or equivalent clinically governed cohorts
- Per-patient calibration and error rates for very mild and mild disease
- Prospective reader studies measuring diagnostic decisions and patient consequences
Living evidence record
Impact record IAI-0U1SS4A
Evidence stage
Studied
Confidence
Supported
Reporting basis
Source analysis
Independent or research support
Present
Record status
Monitoring
Last checked
13 October 2026
Source trail
1 direct source across 1 source type.
People impact
Documented in this record.
Uncertainty
Limits and next checks are explicit.
Stages describe the evidence available—not whether a technology is good or bad. See the public method.
Single-source reporting disclosure
This record analyses one direct source. It can establish what Frontiers in Artificial Intelligence published or reported, but it is not independent corroboration of every performance claim or predicted outcome. The confidence label will change only when broader evidence is added.
Evidence trail
Sources used for this report
Links checked 13 October 2026
Research question, topics and regions
How a four-class Alzheimer's MRI model performs internally and on an independent dataset, and whether its novel classifier head adds measurable value
Topics: Medical imaging · Alzheimer's disease · External validation · Dataset shift · Clinical AI · Model evaluation
Regions: North America · Global
This report is labelled source analysis. We summarise and analyse source material in our own words; company statements remain attributed claims until independently supported. Translated summaries preserve the meaning of the original source and link back to it. Read our editorial standards.
Reader commentary
What does this mean where you are?
Tell us what you have seen at work, in your community or in everyday life. A question is welcome too. No account is needed; we review notes before publishing them.
Explore commentary across the portal →Published reader notes
0No published reader notes yet. You can start the evidence-led discussion above.
Prefer a private correction or response? Contact the newsroom.