In 2022 I wrote an editorial asking whether we were ready for the translation of Machine Learning into clinical settings. Four years later, the question has a new dimension. The European AI Act, Regulation (EU) 2024/1689, now defines in law what a clinical AI system must offer before it can be used on patients. Many of those requirements sound very familiar to anyone working on explainable AI and uncertainty quantification.
This post is not legal advice. I am a researcher, not a lawyer. It is my attempt to translate the parts of the AI Act that matter most for people who build machine learning models on neuroscience data.
Where we are on the timeline
The AI Act entered into force on 1 August 2024 and applies in stages. The bans on prohibited practices have applied since February 2025, and the rules for general-purpose AI models since August 2025. The obligations for high-risk systems were originally due in 2026 and 2027, but the Digital Omnibus on AI postponed them. Standalone high-risk systems now have until 2 December 2027. AI embedded in products covered by sectoral legislation, including medical devices, has until 2 August 2028.

The delay changes the pace, not the destination. The requirements themselves are still there, and two years is not long when you consider how much time it takes to design, validate and document a clinical model properly.
Does it apply to my research?
Mostly not directly. The AI Act excludes AI systems developed and put into service for the sole purpose of scientific research and development. A Random Forest trained on ADNI to study conversion from MCI to Alzheimer’s disease, published in a journal, is not a regulated product.
The picture changes when a model moves toward the clinic. Software that supports diagnosis or clinical decisions usually qualifies as a medical device under the MDR. If it requires a notified body assessment, which is the case for most diagnostic software, it is also a high-risk AI system under the AI Act. In other words, the research we publish today is the prototype of the regulated products of tomorrow. Designing it with those requirements in mind costs much less than retrofitting it later.
Interpretability is now a requirement
Two articles are particularly relevant. Article 13 requires high-risk systems to be transparent enough for the people using them to interpret their output and use it appropriately. Article 14 requires effective human oversight: clinicians must be able to understand the system’s capabilities and limitations, interpret its output correctly, and remain aware of the risk of over-relying on it.
The AI Act does not prescribe SHAP values or glass-box models. It does, however, make one thing clear: a model whose reasoning nobody can follow will struggle to meet these obligations. This is a strong argument for interpretable-by-design approaches, such as Generalized Additive Models and Explainable Boosting Machines. When a black-box model is used, its post-hoc explanations must be stable and meaningful, not just visually appealing.
Over the years we have used Explainable Boosting Machines to predict Alzheimer’s disease from hippocampal subfields (Sarica et al., 2021) and to distinguish Parkinson’s disease from SWEDD (Sarica et al., Brain Imaging and Behavior, 2022), and we recently compared EBM and GAMI-Net on Parkinson’s biomarkers (Camastra et al., 2026). On the stability side, we proposed the rank-biased overlap to measure how much feature importance rankings agree across models (Sarica et al., 2022).
Uncertainty is now a requirement too
Article 15 requires high-risk systems to achieve an appropriate level of accuracy and robustness, and to declare their accuracy metrics in the instructions for use. A single number on a test set is a weak answer to that requirement. A clinician needs to know how reliable this prediction is, for this patient.
This is where uncertainty quantification becomes practical rather than academic. Methods such as conformal prediction attach to each prediction an interval or a set with a guaranteed coverage level. They make the limits of a model explicit, case by case. Combined with explanations, they answer the two questions that matter at the bedside: why did the model say this, and how much should I trust it? This is exactly the idea behind ICeX, our framework for brain age estimation.
In our own work, we combined conformal prediction with quantile regression forests for brain age estimation (Sarica et al., LNCS, 2025), and then integrated it with explainability in the ICeX framework (Sarica et al., Computer Methods and Programs in Biomedicine, 2026).
I discuss explainability and uncertainty quantification together, with examples from neurodegenerative disease assessment, in my recent book chapter with Vera Gramigna.
Data governance and subgroups
Article 10 asks for training, validation and test data that are relevant, sufficiently representative and examined for possible biases. For neuroimaging this has concrete implications: scanner and site effects, the demographic composition of public cohorts, and performance across subgroups such as sex or age. A model that works on average but systematically fails on one subgroup is a problem that should be found in the lab, not in the clinic.
Subgroups are not a regulatory abstraction. When we looked at conversion from mild cognitive impairment to Alzheimer’s disease separately in women and men, the explainable models pointed to different risk profiles (Sarica et al., Brain Sciences, 2024). A single model reported only on average would have hidden that.
What I would do today
- Prefer interpretable models whenever their performance is comparable, and justify the choice when it is not.
- Check the stability of explanations across resamples and methods, instead of reporting a single plot.
- Report uncertainty at the level of the individual prediction, not only aggregate metrics.
- Validate rigorously, with nested cross-validation and, whenever possible, external cohorts.
- Analyse subgroups and report where the model performs worse.
- Document everything: data, preprocessing, model choices and known limitations.
None of this is new. It is simply good methodology. What the AI Act changes is that good methodology is no longer optional for models that aim to reach patients. To me, that is good news.



Leave a Reply