Postdeployment Monitoring and Surveillance Methods, Guidelines, and Possibilities for AI in Radiology
Vasantha Kumar Venugopal; Suyash Anil Khubchandani; Charlene Liew; Felipe Kitamura; Rohit Takhar; Gerald Lip
RadioGraphics - Volume 46, Number 7 - https://doi.org/10.1148/rg.250173
Summary
As radiology AI systems move from predeployment validation to routine radiology practice, attention is shifting toward postdeployment monitoring and postmarket surveillance in a total product life cycle (TPLC) paradigm. In an operational sense, the human clinical oversight can be positioned along a spectrum encompassing human-in-the-loop (HITL), human-in-a-parallel loop (HIPL), human-on-the-loop (HOTL), human-over-the-loop (HOVL), and human-out-of-the-loop (HOOTL) models. Each of these models offers a trade between the verification workload and autonomy and risk. The authors provide a deeper understanding of the definitions first and then present HOTL as a pragmatic model of human-AI oversight for high-stakes imaging, demonstrating a balance between the trade-offs and benefits. The proposed monitoring system is based on two families of data points that do not require immediate determination of the ground truth, namely temporal stability of inputs and outputs and predictive divergence relative to a deployment initiation baseline. The authors also bring uncertainty quantification into the fray as a third element in helping prioritize reviews when labels are delayed or not continually feasible. The described threshold-based alerting system is paired with tiered escalation mechanisms and root cause analysis to distinguish degradation of the AI algorithm from data or integration pipeline issues. The result is an education-first proactive road map for postdeployment monitoring that allows preservation and prioritization of patient safety while enabling responsible scaling of radiology AI. © RSNA, 2026 See the invited commentary by Rouzrokh and Rouzrokh in this issue.
Figure 1: Conceptual depiction of an RCA workflow for AI monitoring alerts in a proposed HOTL system. env. = environment.
Figure 2: Graph shows predictive divergence (Jensen-Shannon divergence, weekly rolling divergence scores) for studies from a single scanner. A software update in week 6 preceded a threshold breach in week 7, prompting an RCA and a scoped rollback during week 8 as depicted by the arrow. During this period, the use of the triaging AI algorithm was paused while the errors were fixed in preprocessing. After retesting, the use of the AI algorithm was reinstated with divergence scores then reverting back to normal in week 10.
Keywords:
MEDLINE | Product (mathematics) | Interventional radiology | Patient safety
References
1. Hill DLG. AI in imaging: the regulatory landscape. Br J Radiol 2024;97(1155):483–491.
2. Topol EJ. High-performance medicine: the convergence of human and artificial intelligence. Nat Med 2019;25(1):44–56.
3. Wiens J, Saria S, Sendak M, et al. Do no harm: a roadmap for responsible machine learning for health care. Nat Med 2019;25(9):1337–1340. [Published correction appears in Nat Med 2019;25(10):1627.]
4. Kelly CJ, Karthikesalingam A, Suleyman M, Corrado G, King D. Key challenges for delivering clinical impact with artificial intelligence. BMC Med 2019;17(1):195.
5. McKinney SM, Sieniek M, Godbole V, et al. International evaluation of an AI system for breast cancer screening. Nature 2020;577(7788):89–94. [Published correction appears in Nature 2020;586(7829):E19.]
6. de Vries CF, Colosimo SJ, Staff RT, et al; iCAIRD Radiology Collaboration. Impact of different mammography systems on artificial intelligence performance in breast cancer screening. Radiol Artif Intell 2023;5(3):e220146.
7. Sahiner B, Chen W, Samala RK, Petrick N. Data drift in medical machine learning: implications and potential remedies. Br J Radiol 2023;96(1150):20220878.
8. Roschewitz M, Khara G, Yearsley J, et al. Automatic correction of performance drift under acquisition shift in medical image classification. Nat Commun 2023;14(1):6608.
9. Floridi L, Cowls J, Beltrametti M, et al. AI4People—An ethical framework for a good AI society: opportunities, risks, principles, and recommendations. Minds Mach (Dordr) 2018;28(4):689–707.
10. Cabitza F, Rasoini R, Gensini GF. Unintended consequences of machine learning in medicine. JAMA 2017;318(6):517–518.
11. U.S. Food and Drug Administration. Artificial Intelligence/Machine Learning (AI/ML)-Based Software as a Medical Device (SaMD) Action Plan. https://www.fda.gov/media/145022/download . Published January 2021. Accessed August 18, 2025.
12. U.S. Food and Drug Administration. Predetermined Change Control Plans for Machine Learning-Enabled Medical Devices: Guiding Principles. https://www.fda.gov/medical-devices/software-medical-device-samd/predetermined-change-control-plans-machine-learning-enabled-medical-devices-guiding-principles . Published October 2022. Accessed August 18, 2025.
13. Brady AP, Allen B, Chong J, et al. Developing, Purchasing, Implementing and Monitoring AI Tools in Radiology: Practical Considerations. A Multi-Society Statement from the ACR, CAR, ESR, RANZCR and RSNA. Radiol Artif Intell 2024;6(1):e230513.
14. Ross J, Hammouche S, Chen Y, et al; Royal College of Radiologists AI Working Group. Beyond regulatory compliance: evaluating radiology artificial intelligence applications in deployment. Clin Radiol 2024;79(5):338–345.
15. Dolin P, Li W, Dasarathy G, Berisha V. Statistically Valid Post-Deployment Monitoring Should Be Standard for AI-Based Digital Health. arXiv 2025. Preprint posted online June 6, 2025. https://arxiv.org/abs/2506.05701 .
16. Strickland NH. Quality assurance in radiology: peer review and peer feedback. Clin Radiol 2015;70(11):1158–1164.
17. Daye D, Wiggins WF, Lungren MP, et al. Implementation of Clinical Artificial Intelligence in Radiology: Who Decides and How? Radiology 2022;305(3):555–563. [Published correction appears in Radiology. 2022 Oct;305(1):E62.]
18. Oala L, Suresh H, Madras D, et al. Monitoring machine learning models in healthcare. In: Proceedings of the NeurIPS 2020 Workshop on Machine Learning for Health (ML4H). 2020.
19. Dombrowski M, Prenner A, Kainz B. Bias Assessment and Data Drift Detection in Medical Image Analysis: A Survey. arXiv 2024. Preprint posted online September 26, 2024; doi:.
20. Koch LM, Baumgartner CF, Berens P. Distribution shift detection for the postmarket surveillance of medical AI algorithms: a retrospective simulation study. NPJ Digit Med 2024;7(1):120.
21. Roschewitz M, Mehta R, Jones C, Glocker B. Automatic dataset shift identification to support root cause analysis of AI performance drift. arXiv 2024. Preprint posted online November 12, 2024. https://arxiv.org/abs/2411.07940 .
22. Dratsch T, Chen X, Rezazade Mehrizi M, et al. Automation Bias in Mammography: The Impact of Artificial Intelligence BI-RADS Suggestions on Reader Performance. Radiology 2023;307(4):e222176.
23. Patel BN, Rosenberg L, Willcox G, et al. Human-machine partnership with artificial intelligence for chest radiograph diagnosis. NPJ Digit Med 2019;2:111. [Published correction appears in NPJ Digit Med 2019;2:129.]
24. U.S. Food and Drug Administration (FDA). Artificial Intelligence-Enabled Device Software Functions: Lifecycle Management and Marketing Submission Recommendations. https://www.fda.gov/regulatory-information/search-fda-guidance-documents/artificial-intelligence-enabled-device-software-functions-lifecycle-management-and-marketing-submission-recommendations . Published online January 2025. Accessed August 18, 2025.
25. Agência Nacional de Vigilância Sanitária (ANVISA). Resolução de Diretoria Colegiada – RDC No. 657, de 24 de março de 2022: Dispõe sobre Software como Dispositivo Médico (SaMD). https://in.gov.br/en/web/dou/-/resolucao-de-diretoria-colegiada-rdc-n-657-de-24-de-marco-de-2022-389603457 . Published online March 2022. Accessed August 18, 2025.
26. European Union. Regulation (EU) 2017/745 on medical devices. https://eur-lex.europa.eu/legal-content/EN/TXT/?uri=CELEX:32017R0745 . Published online 2017. Accessed August 18, 2025.
27. Health Canada. Draft Guidance on Pre-market Requirements for Machine Learning-enabled Medical Devices. https://www.canada.ca/en/health-canada/services/drugs-health-products/medical-devices/activities/guidance-documents/draft-guidance-machine-learning.html . Published online March 2023. Accessed August 18, 2025.
28. Medicines and Healthcare products Regulatory Agency (MHRA). Software and AI as a Medical Device Change Programme. https://www.gov.uk/government/publications/software-and-ai-as-a-medical-device-change-programme . Published online October 2023. Accessed August 18, 2025.
29. Pharmaceuticals and Medical Devices Agency (PMDA). Basic Concept of the Evaluation of Medical Device Programs Using Artificial Intelligence. https://www.pmda.go.jp/english/safety/info-services/ai-healthcare/0021.html . Published online June 2022. Accessed August 18, 2025.
30. Senado Federal do Brasil. Projeto de Lei n\textsuperscripto 2338. de 2023: Estabelece princípios, direitos e deveres para o uso da inteligência artificial no Brasil. https://www25.senado.leg.br/web/atividade/materias/-/materia/157233 . Published online 2023. Accessed August 18, 2025.
31. Singapore Health Sciences Authority (HSA). AI in Healthcare Guidelines (AIHGle). https://www.hsa.gov.sg/ai-in-healthcare . Published online 2022. Accessed August 18, 2025.
32. Venugopal VK, Gupta A, Takhar R, Mahajan V. New Epochs in AI Supervision: Design and Implementation of an Autonomous Radiology AI Monitoring System. arXiv 2023. Preprint posted online November 24, 2023. doi:.
33. Chen M, Wang Y, Wang Q, et al. Impact of human and artificial intelligence collaboration on workload reduction in medical image interpretation. NPJ Digit Med 2024;7(1):349.
34. Davenport T, Kalakota R. The potential for artificial intelligence in healthcare. Future Healthc J 2019;6(2):94–98.
35. Gawlikowski J, Tassi CRN, Ali M, et al. A survey of uncertainty in deep neural networks. Artif Intell Rev 2023;56(S1):1513–1589.
36. Benjamin M, Engelhard G, Aisen A, Aradi Y, Benjamin E. A Multisite, Report-Based, Centralized Infrastructure for Feedback and Monitoring of Radiology AI/ML Development and Clinical Deployment. arXiv 2020. Preprint posted online August 31, 2020. https://arxiv.org/abs/2008.13781 .
37. Integrating the Healthcare Enterprise (IHE) International. AI Interoperability in Imaging: IHE Radiology White Paper (Revision 1.1). https://www.ihe.net/resources/technical_frameworks/#Radiology . Published online October 2021. Accessed August 18, 2025.
38. Cummings ML. What self-driving cars tell us about AI risks. IEEE Spectrum . https://spectrum.ieee.org/self-driving-cars-2662494269 . Published July 30, 2023. Accessed August 20, 2025.
39. Chow J, Lee R, Wu H. How do radiologists currently monitor AI in radiology and what challenges do they face? An interview study and qualitative analysis. J Imaging Inform Med 2025. 10.1007/s10278-025-01493-8 . Published online April 8, 2025.
40. Abdar M, Pourpanah F, Hussain S, et al. A review of uncertainty quantification in deep learning: Techniques, applications and challenges. Inf Fusion 2021;76:243–297.
41. Faghani S, Moassefi M, Rouzrokh P, et al. Quantifying Uncertainty in Deep Learning of Radiologic Images. Radiology 2023;308(2):e222217.
42. Royal College of Physicians. National Early Warning Score (NEWS) 2: Standardising the Assessment of Acute-Illness Severity in the NHS. Royal College of Physicians ; 2017.
43. Thor J, Lundberg J, Ask J, et al. Application of statistical process control in healthcare improvement: systematic review. Qual Saf Health Care 2007;16(5):387–399.
44. Benneyan JC, Lloyd RC, Plsek PE. Statistical process control as a tool for research and healthcare improvement. Qual Saf Health Care 2003;12(6):458–464.
45. Liew CJY. Autonomous algorithmic monitoring of a deep-learning chest radiograph AI model using temporal divergences in a real-world clinical setting. Abstract RPS 1205:276 In: European Congress of Radiology (ECR) 2024; March 1, 2024; Vienna, Austria. https://connect.myesr.org/course/recent-development-in-ai-for-lung-nodule-detection/ .
46. National Institute of Standards and Technology (NIST). AI Risk Management Framework (AI RMF 1.0) Playbook. https://airc.nist.gov/airmf-resources/playbook/ . Published online 2023. Accessed August 18, 2025.
47. Mello-Thoms C, Mello CAB. Clinical applications of artificial intelligence in radiology. Br J Radiol 2023;96(1150):20221031.
48. Erdal BS, Dikici E, Bigelow M, Prevedello LM, White RD. Integrating AI into Radiology Workflow: Levels of Research, Production, and Feedback Maturity. arXiv 2019. Preprint posted online October 14, 2019. https://arxiv.org/abs/1910.06424 .
49. Cardoso MJ, Moosbauer J, Cook TS, et al. RAISE -- Radiology AI Safety, an End-to-end lifecycle approach. arXiv 2023. Preprint posted online November 24, 2023; doi:.
50. World Health Organization. Regulatory Considerations on Artificial Intelligence for Health. https://www.who.int/publications/i/item/9789240074148 . Published 2023. Accessed August 18, 2025.
51. U.S. Department of Commerce. National Institute of Standards and Technology. AI Risk Management Framework (AI RMF 1.0) . https://doi.org/10.6028/NIST.AI.100-1 . Published 2023. Accessed August 20, 2025.
52. Gichoya JW, Banerjee I, Bhimireddy AR, et al. AI recognition of patient race in medical imaging: a modelling study. Lancet Digit Health 2022;4(6):e406–e416.
53. Busch F, Kather JN, Johner C, et al. Navigating the European Union Artificial Intelligence Act for Healthcare. NPJ Digit Med 2024;7(1):210.
54. Health Sciences Authority (Singapore). Public Consultation on the Proposed Exemption from Manufacturer’s Licensing and Product Registration Requirements for Artificial Intelligence–Software as a Medical Device Developed by Selected Public Healthcare Entities for Use in Public Healthcare. https://www.hsa.gov.sg/announcements/public-consultation/proposedexemption-AI-SaMD . Published online May 19, 2025. Accessed August 20, 2025.
Other resources
Diffusion-Weighted Imaging with Color-Coded Images: Towards a Reduction in Reading Time While Keeping a Similar Accuracy
The aim of this study was to develop a diagnostic tool capable of providing diffusion and apparent diffusion coefficient (ADC) map information in a single color-coded image and to assess the performance of color-coded images compared with their corresponding diffusion and ADC map. The institutional review board approved this retrospective study, which sequentially enrolled 36 head MRI scans. Diffusion-weighted images (DWI) and ADC maps were compared to their corresponding color-coded images. Four raters had their interobserver agreement measured for both conventional (DWI) and color-coded images. Differences between conventional and color-coded images were also estimated for each of the 4 raters. Cohen’s kappa and percent agreement were used. Also, paired-samples t -test was used to compare reading time for rater 1. Conventional and color-coded images had substantial or almost perfect agreement for all raters. Mean reading time of rater 1 was 47.4 seconds for DWI and 27.9 seconds for color-coded images ( P = .00007). These findings are important because they support the role of color-coded images as being equivalent to that of the conventional DWI in terms of diagnostic capability. Reduction in reading time (which makes the reading easier) is also demonstrated for one rater in this study.
The RSNA Pediatric Bone Age Machine Learning Challenge
Purpose The Radiological Society of North America (RSNA) Pediatric Bone Age Machine Learning Challenge was created to show an application of machine learning (ML) and artificial intelligence (AI) in medical imaging, promote collaboration to catalyze AI model creation, and identify innovators in medical imaging. Materials and Methods The goal of this challenge was to solicit individuals and teams to create an algorithm or model using ML techniques that would accurately determine skeletal age in a curated data set of pediatric hand radiographs. The primary evaluation measure was the mean absolute distance (MAD) in months, which was calculated as the mean of the absolute values of the difference between the model estimates and those of the reference standard, bone age. Results A data set consisting of 14 236 hand radiographs (12 611 training set, 1425 validation set, 200 test set) was made available to registered challenge participants. A total of 260 individuals or teams registered on the Challenge website. A total of 105 submissions were uploaded from 48 unique users during the training, validation, and test phases. Almost all methods used deep neural network techniques based on one or more convolutional neural networks (CNNs). The best five results based on MAD were 4.2, 4.4, 4.4, 4.5, and 4.5 months, respectively. Conclusion The RSNA Pediatric Bone Age Machine Learning Challenge showed how a coordinated approach to solving a medical imaging problem can be successfully conducted. Future ML challenges will catalyze collaboration and development of ML tools and methods that can potentially improve diagnostic accuracy and patient care. © RSNA, 2018 Online supplemental material is available for this article. See also the editorial by Siegel in this issue.
The Type III Secretion System (T3SS)-Translocon of Atypical Enteropathogenic Escherichia coli (aEPEC) Can Mediate Adherence
The intimin protein is the major adhesin involved in the intimate adherence of atypical enteropathogenic Escherichia coli (aEPEC) strains to epithelial cells, but little is known about the structures involved in their early colonization process. A previous study demonstrated that the type III secretion system (T3SS) plays an additional role in the adherence of an Escherichia albertii strain. Therefore, we assumed that the T3SS could be related to the adherence efficiency of aEPEC during the first stages of contact with epithelial cells. To test this hypothesis, we examined the adherence of seven aEPEC strains and their eae (intimin) isogenic mutants in the standard HeLa adherence assay and observed that all wild-type strains were adherent while five isogenic eae mutants were not. The two eae mutant strains that remained adherent were then used to generate the eae/escN double mutants (encoding intimin and the T3SS ATPase, respectively) and after the adherence assay, we observed that one strain lost its adherence capacity. This suggested a role for the T3SS in the initial adherence steps of this strain. In addition, we demonstrated that this strain expressed the T3SS at significantly higher levels when compared to the other wild-type strains and that it produced longer translocon-filaments. Our findings reveal that the T3SS-translocon can play an additional role as an adhesin at the beginning of the colonization process of aEPEC.