AI & Digital Health · 4 min read
Explainable AI in Surgery: Why Transparency Is Not Optional
A clinical AI system that cannot explain its reasoning is not a clinical decision support tool. It is an oracle — and medicine has no place for oracles. Explainability is not a technical nicety in healthcare AI. It is the prerequisite for the critical engagement that patient safety requires from every clinician who uses a digital decision support tool.
The tension between model performance and model explainability is one of the foundational challenges of clinical AI. The most accurate predictive models — deep learning architectures with millions of parameters — are also the most opaque: their outputs cannot be traced to specific, interpretable features in the way that a logistic regression coefficient or a decision tree branch can be. This opacity is acceptable in domains where outputs can be verified independently of the reasoning process. In clinical medicine, it is not.
Why explainability matters clinically
A clinician who receives an AI recommendation — "this patient has a 73% probability of requiring revision arthroplasty within five years" — and cannot understand what features of the patient drove that prediction cannot critically evaluate whether the prediction is appropriate for this patient, whether the model's assumptions match the clinical context, or whether known limitations of the model are relevant to this case. Without that understanding, the recommendation cannot be meaningfully accepted or appropriately overridden. The clinician is reduced to either blind acceptance or arbitrary rejection — neither of which constitutes clinical judgment.
Explainability methods — SHAP values, LIME, attention maps, concept-based explanations — provide varying degrees of insight into what features a model uses to generate predictions. SHAP values can decompose a model's prediction into the contribution of each input feature, presented in clinical language: "this patient's age, BMI, and pre-operative PROM score together account for 60% of the elevated revision risk prediction." This is actionable information — it tells the clinician what to verify, what to question, and what the model's reasoning is based on.
The regulatory direction of travel
The EU AI Act classifies clinical decision support AI as high-risk, requiring transparency, human oversight, and documentation of the system's capabilities and limitations. The FDA's Digital Health Center of Excellence and the MHRA's Software as a Medical Device framework both move in the same direction: explainability and transparency as regulatory requirements, not optional design features. Vendors who cannot explain what their models are doing are building clinical AI that is increasingly difficult to deploy legally, and impossible to use responsibly.
Explainability is the bridge between the model's output and the clinician's judgment. Without it, AI in clinical medicine is not decision support. It is decision replacement — and that is a patient safety risk, not a clinical advance.
💬 Do the AI tools you use in clinical practice provide explainable outputs — and has the quality of explanation ever changed whether you accepted or overrode a recommendation?