global

Variables

Utilities

CUSTOM STYLES

Clinical, Cultural, Computational, and Regulatory Considerations to Deploy AI in Radiology: Perspectives of RSNA and MICCAI Experts

Authors:

Marius George Linguraru; Spyridon Bakas; Mariam Aboian; Peter Chang; Adam E. Flanders; Jayashree Kalpathy–Cramer; Felipe Kitamura; Matthew P. Lungren; John Mongan; Luciano M. Prevedello; Ronald M. Summers; Carol C. Wu; Maruf Adewole; Charles E. Kahn

Publication

Radiology: Artificial Intelligence - Volume 6, Number 4 - https://doi.org/10.1148/ryai.240225

Download PDF

Summary

The Radiological Society of North of America (RSNA) and the Medical Image Computing and Computer Assisted Intervention (MICCAI) Society have led a series of joint panels and seminars focused on the present impact and future directions of artificial intelligence (AI) in radiology. These conversations have collected viewpoints from multidisciplinary experts in radiology, medical imaging, and machine learning on the current clinical penetration of AI technology in radiology and how it is impacted by trust, reproducibility, explainability, and accountability. The collective points—both practical and philosophical—define the cultural changes for radiologists and AI scientists working together and describe the challenges ahead for AI technologies to meet broad approval. This article presents the perspectives of experts from MICCAI and RSNA on the clinical, cultural, computational, and regulatory considerations—coupled with recommended reading materials—essential to adopt AI technology successfully in radiology and, more generally, in clinical practice. The report emphasizes the importance of collaboration to improve clinical deployment, highlights the need to integrate clinical and medical imaging data, and introduces strategies to ensure smooth and incentivized integration. Keywords: Adults and Pediatrics, Computer Applications–General (Informatics), Diagnosis, Prognosis © RSNA, 2024

Key points

Artificial intelligence (AI) tools can play a key role in radiology if radiologists trust in their design, deploy them with adequate training, and establish clear guidelines regarding clinical accountability.
For successful AI deployment, radiologists and AI scientists should define a unified agenda, language, and set of expectations.
Clinical institutions should align their staffing, data flow, and computational resources to deploy and monitor AI systems.

Introduction

This special report is the third in a series of articles based on joint panels and seminars by members of the Radiological Society of North America (RSNA) and the Medical Image Computing and Computer Assisted Intervention (MICCAI) Society that focus on the present impact and future directions of artificial intelligence (AI) in radiology ¹^,². The current report addresses the clinical deployment of AI into the practice of diagnostic radiology and explores the clinical, cultural, computational, and regulatory considerations that imaging experts, clinical and technical, should consider for successful adoption of AI technology. Recommended reading relevant to each section is provided in the Table in the article. This report also introduces strategies to ensure smooth and incentivized integration of AI into the radiology workflow.

Clinical Considerations

Tools to exchange data either for centralized analytics or distributed learning will be able to support sharing of data and/or models but will require additional institutional infrastructure and support.

Image annotation may become a less substantial hurdle in the next 5 to 10 years.

AI models built with small datasets or in-house data can navigate challenges related to data bias and diversity if they are applied to targeted populations and carefully monitored.

Data Sharing and Analytics

Data sharing is crucial to foster the development of machine learning (ML) models. Many institutions are not allowed or are unwilling to share their data, and there is a critical shortage of data on rare diseases ³. Clinicians may feel uncomfortable sharing images because of lack of time and expertise to extract, anonymize, and upload large volumes of data efficiently. Thus, data sharing can limit the deployment of AI into clinical practice and can pose a substantial barrier between institutions and even within the same institution. Data sharing also must abide by ethical and legal standards ⁴^,⁵.

When institutions are open to data sharing, methods for data collection and transfer must be secure. Email is insecure and inefficient, compact discs are time-intensive and require physical mailing, and hard drives require exchange of physical media. File-sharing platforms are more secure than email but can be slow and require multi-institutional information technology (IT) approval for external collaborations. A centralized research picture archiving and communication system (PACS) can be expensive and can have a limited array of data-analytic tools. Open-source and commercial platforms are available that incorporate AI, but tools are still developing. Cloud-based servers with AI capabilities require expertise to make sure data are secure and Health Insurance Portability and Accountability Act compliant. For institutions that are not open to data sharing, federated learning can facilitate AI model development and deployment without data exchange ⁶^–⁸, but federated learning facilities are not commonly available in radiology departments and require specialized expertise ⁹.

Data Annotation

AI relies on well-curated and annotated data. Annotation is time-consuming and labor-intensive and poses challenges for large datasets ¹⁰. Multiple annotators are required to tackle large datasets and allow measurement of interrater variability. The RSNA AI challenges have shown that expert annotators can exhibit substantial variability even when the annotation tasks are well-defined, highly documented, and carefully communicated ¹¹^–¹⁴. Thus, it is important to record and make available all details of the annotation process, including methods for adjudication and quality assurance, to those who are evaluating AI tools for clinical deployment.

Annotation of radiologic images often requires transferring images to a research computer that has annotation software, potentially requiring uploads, and only then starting the segmentation or other type of annotation. Some institutions have been able to work with their IT departments to directly transfer data from the clinical PACS to the annotation software ¹⁵, but most do not have such capabilities. Thus, a major hurdle for annotation is the lack of institutional infrastructure for data access and sharing with annotators. The availability of web-based, easy-to-use annotation tools allows radiologists from around the world to contribute their expertise and is key to reducing hurdles in annotations once the de-identified data can be secured from the contributing institutions.

On the other hand, there has been substantial evolution in our understanding of annotation and the level of quality required for AI development ¹⁶. Although very high-quality testing data are important, noise can be tolerated in the training data, providing it does not contain bias. New technologies for active learning, iterative annotation, and self-supervised learning enable AI models to learn more rapidly from fewer annotations.

Bias and Model Performance

ML models often learn biases in the training data. Underrepresented characteristics, such as image acquisition features or patient features, can be especially prone to bias. Potential sources of bias differ for various AI applications and depend on the clinical setting in which the AI tool will be used. Diversity of background and experience in both the development and deployment teams is helpful to identify and address biases. Because it is not feasible for an AI model to work everywhere reliably, it is important to recognize the intended use and potential limitations of the model ¹⁷^–¹⁹. For instance, an institution may want to develop a model for in-house use only. In that case, it will be practical to develop a model that performs well only on internal data. However, the bias and limitation of the model, likely not applicable to underrepresented populations in research, must be clearly disclosed. It is critical to continuously monitor performance, identify changes in performance, and provide detailed and updated documentation.

Cultural Considerations

Training radiologists to work with AI tools will become routine.

Motivation for radiologists to use AI is key to increasing clinical efficiency and to allowing them to perform complex tasks, such as measuring volumes, predicting outcomes, and integrating imaging data with the broader electronic health record (EHR).

AI tools must be designed with the trust of radiologists and clear definitions of clinical accountability.

Volumetrics

Automation and AI in radiology potentially can provide routine three-dimensional (3D) volume measurements, which represents a cultural shift from two-dimensional (2D) measurements such as the Response Evaluation Criteria in Solid Tumors (ie, RECIST) ²⁰^–²². For most radiologists, qualitative image interpretation and 2D measurements are still the standard of care in clinical practice and in radiology reports. Studies have shown the benefits of volumetric measurements, such as in cancer imaging, where 3D volumetrics offer better outcome prediction and treatment response evaluation than qualitative assessment or 2D measurements ²³^–²⁷. Without AI, such measurements require a substantial amount of additional time. To prepare radiologists for volumetric data interpretation, radiology training should emphasize a shift to 3D from 2D measurement, and dedicated education on lesion contouring should be incorporated into radiology training.

Many radiologists and other specialists would like to have better 3D measurement tools. However, few 3D tools are available at every radiologist’s desktop, although there is an increased availability of contouring tools in clinical PACS. In addition to contouring tools, there is a critical need for tools that allow individual lesion tracking, for example, in the setting of metastatic disease. Thus, there is increasing potential for volumetric tools—powered by AI and easily adjustable by radiologists—to play a key role in clinical care.

Image Evaluation to Outcome Prediction

Outcome prediction is a valuable end goal. AI-based prediction of cancer mortality, cardiovascular morbidity, or organ function, among others, can be competitive with routinely available clinical methods ²⁸^–³⁰. Outcome prediction also may motivate radiologists to focus on data interpretation with the help of AI. To be clinically relevant, outcome prediction needs to be performed in a clinical scenario where it has the potential to change care. There are risks in outcome prediction, too. Models may be biased in ways that human experts cannot appreciate easily ³¹. If the task is image evaluation, it is more likely that a human reader would catch the error. But if the model is expected to have superhuman performance (say 5-year risk prediction for a task that a human cannot do), the model deployment must be carefully monitored to alleviate risks.

Human-Machine Interaction

In daily practice, radiologists’ measurements can exhibit substantial intraobserver and interobserver variability ³²^–³⁴. AI-based algorithms may reduce measurement variability and may allow greater consistency. However, automation bias can increase error rates in the setting of AI utilization, and human-AI interactions are understudied. Even if an AI model works well, there is no guarantee radiologists will use it correctly to improve clinical performance. Human-AI interactions and their cultural and clinical implications require further research to ensure optimal performance ³⁵. The design of stronger and more comprehensive programs to train radiologists to work with AI approaches is important, and these programs should be integrated into early career curriculum ³⁶^–³⁸.

Clinical Integration

Radiology AI systems can be integrated with currently used software to reduce the effort of interpretation and reporting ³⁹. Ideally, AI should reduce the effort of interpretation and reporting. A major characteristic of radiologists’ daily workflow is the long list of studies that need to be read every day, which makes the radiologist prioritize efficiency in reading studies. Any activities outside of reading can be a substantial distraction.

AI tools within PACS provide an efficient method for radiologists to adapt them into clinical practice. Integration of AI into PACS allows manual corrections of segmentations that would not be possible otherwise, thus providing another level of safety checks. AI tools can prioritize worklists: They can identify studies with acute findings or automatically route studies with specific pathology to the best-matched specialist radiologists ⁴⁰.

An important challenge in radiology is the limited clinical information provided when a study is being interpreted. Although an EHR is available, its organization may not allow one to extract information easily. Therefore, integration of EHRs with AI tools in radiology has great potential for clinical practice ⁴¹. AI tools that extract and summarize clinically relevant information from the EHR can support more precise interpretation of studies and improve radiology reports.

Trust and Accountability

Trust is important in clinical practice. To assure patient safety, the field of medicine is more conservative than others in adopting new technology ⁴². Moreover, AI applications are not always intuitive, and “black box” models that lack explainability are difficult to trust. As such, few clinicians trust AI for autonomous evaluation, and AI deployment strategies must incorporate radiologist supervision for the near future. Radiologists are, for the most part, comfortable to be held accountable for studies they interpret with or without AI support ⁴³. For example, if the AI tool highlights a positive finding, the radiologist can visualize and confirm the finding with confidence. Exceptions to explainability should be considered, such as for identification of radiomics features that cannot be perceived by the human eye.

An integrated approach to build trust should include retrospective validation with local data (ie, test on your own data to trust it) and prospective evaluation using AI in practice. The latter is tied to having high positive and negative predictive values. Because algorithms are not error-proof, it is important for radiologists to understand how errors may happen and that they are something to be expected. To earn the trust of clinicians, AI algorithms also must support the clinical workflow (eg, automatic measurements incorporated into radiology reports) and expedite the turnaround of studies. AI methods where model predictions are not repeatable, do not generalize within the scope of their development, or are not well calibrated should not be implemented. These and other concerns on accountability, quality control, and trust have been addressed in publications by leading experts from international societies including RSNA, MICCAI, and the American Association of Physicists in Medicine ⁴⁴^,⁴⁵.

Computational Considerations

Cloud computing may be most effective for radiology departments that lack hardware and maintenance resources.

The design, development, deployment, and monitoring of radiology AI tools should be done by data scientists and radiologists working together.

Clinical institutions must prioritize the data flow from acquisition devices to PACS and/or data lakes and to AI servers.

Hardware Availability

To deploy AI in clinical settings requires consideration of computational factors for training and inference, such as hardware availability ⁴⁶. In most clinical centers, and especially in limited-resource settings, the cost and maintenance of graphics processing unit servers and high-performance computing resources limits their availability ⁴⁷. Approaches such as cloud-based systems may be necessary. The use of commercial algorithms in the cloud or homegrown algorithms on-premises can also present challenges. However, if graphics processing unit memory requirements can be reduced for AI models during inference, hardware availability should not be a substantial challenge. Overall, although hardware availability may depend on the location and use case, less expensive central processing units may be sufficient for many AI applications.

AI Scientists in Radiology Departments

Communication between radiologists and AI scientists is key to incorporate AI into clinical practice. Radiologists are highly skilled at characterizing phenotypes and labeling images but often lack the tools to annotate datasets and build databases ⁴⁸. AI scientists, on the other hand, have skills in image processing and algorithm development, but they often focus on questions that may not have the highest yield in clinical practice. It is crucial to encourage communication between radiologists and AI scientists through grant mechanisms, conferences, and professional and social gatherings to build trust and collaboration.

The shortage of skilled AI scientists in radiology departments poses another limitation. Academic medical centers frequently compete with technology companies for data scientists. To create robust and reliable AI models, it is important to have AI/data scientists, software engineers, PACS/radiology information system specialists, and radiologists working together ²^,⁴⁹.

AI Performance

In general, clinical accuracy is more important than speed ⁵⁰. If inference time is a substantial problem, one can limit the model’s size to not exceed a given time per inference and train and tune that model to the best performance possible. In many applications, speed is not particularly relevant: A computation time of a few minutes generally will not be noticeable. One exception is for tasks such as interactive segmentation, where the user must wait for the AI system to respond. As algorithms increase in accuracy with the availability of larger training datasets, it will be possible in many scenarios to use less complex AI models. Less complex models will lead to more rapid computation and distribution of AI results to the clinic and the need for a framework to evaluate their performance ⁵¹. In many clinical settings, the primary bottleneck is not computational time, but rather the time to transfer data from acquisition device (eg, CT scanner) to PACS to AI inference server.

Radiology practice is fast-paced, dynamic, and often stressful; it requires algorithms that can keep up with the demands of clinical practice. AI algorithms can be incorporated into practice automatically or activated manually by a radiologist ⁵². Although an algorithm’s accuracy is important, consistency of measurement is its inherent advantage ⁵³. The qualities of an algorithm need to be evaluated in the context of the clinical problem and use case. Domain shift and algorithm generalizability remain challenging questions for research and clinical implementation.

Continuous Evaluation

Continuous evaluation and quality control of AI tools is a critical aspect of clinical implementation ⁵⁴. Monitoring of performance is not trivial and is currently a weakness for most AI systems. Centralized datasets that can be used as “phantoms” to test algorithms and image processing software may provide one solution. Technologies for continuous evaluation must not require explicit supervision or labels. Therefore, regulatory guidelines should assure continuous evaluation ⁵⁵. Most organizations lack the staff to perform these evaluations effectively. For models with access to reference standard data, continuous evaluation and retraining could be automated. For models without such data, expensive and time-consuming annotation could be limited by focusing validation on new images in the event of a scanner replacement or protocol change.

Regulatory Considerations

Radiology is the leading application field of U.S. Food and Drug Administration (FDA)–approved medical AI devices.

Vision and language models can positively impact the field of radiology and should undergo regulatory oversight.

The role of regulatory approval in the implementation of AI tools in clinical practice will require ongoing consideration.

Regulatory Guidelines

The field of medical AI is rapidly evolving, and regulatory agencies such as the FDA are striving to keep up with the pace of innovation ⁵⁶. Notably, the development of large foundation and generative AI models is progressing fast, including vision and language models, which are AI tools that integrate understanding of images and text. Nevertheless, technology development outpaces the creation of regulatory guidelines; as of October 2023, there was no FDA-authorized device that used generative AI ⁵⁷. As a result, commercial development favors some types of algorithms over others. For example, the computer-aided triage pathway has seen relatively high rates of FDA clearance, which has resulted in a flood of worklist prioritization tools in the market. Similarly, time-limited financial incentives such as new technology add-on payment (ie, NTAP) reimbursement have driven demand for tools in the hope of an early financial windfall that has largely been exaggerated. There is also concern that many FDA-approved algorithms have not undergone sufficient external or prospective validation. Published reports of performance may not accurately reflect how well these algorithms perform in real-world settings ⁵⁸.

Regulatory Approval

Regulatory authorization is critical for the rollout of AI tools in clinical practice ⁵⁹. By July 2023, 79% of the medical AI devices authorized in 2023 by the FDA were in radiology, followed by cardiology and neurology ⁵⁷. There is support for the authorization of software as a medical device, which allows for innovation and does not stifle progress. FDA clearance is not required to use locally developed AI tools; it focuses on regulating marketing claims and commercialization. Moreover, marketing claims and FDA submission materials are not always concordant. However, strong oversight and understanding by practitioners of potential harms, particularly in vulnerable populations, is important. Regulatory approval is needed for potential financial reimbursement for the use of AI and facilitating the transition from research to clinical practice.

Outlook

Large and general foundation and generative AI models, including vision and language models, can impact the clinical adoption of AI tools and may help reduce burnout in radiology.

Financial incentives to use AI will encourage hospitals to invest in AI-based software and increase the motivation for clinicians to use new technologies.

To advance radiology AI tools in clinical care and research, multidisciplinary societies can adopt a unified agenda, language, and set of expectations.

Impact of the COVID-19 Pandemic

The COVID-19 pandemic has had a mixed impact on the clinical adoption of AI ⁶⁰. On the positive side, the intellectual and clinical challenges of this new, dangerous, and highly contagious disease spurred radiologists and AI scientists around the world to create algorithms for diagnosis and triage ⁶¹^,⁶² and to establish new data repositories like the Medical Imaging and Data Resource Center (ie, MIDRC) ⁶³. The increase in imaging volume, coupled with the pre-existing shortage of radiologists, made radiologist burnout a more widespread issue and encouraged radiologists to explore AI tools to increase efficiency and accuracy in clinical practice. The desire to quickly integrate COVID risk models yielded an expedited pathway to move tools from research into the clinical realm. The pandemic also presented barriers to the clinical adoption of AI, as new software could not be implemented easily due to restrictions on travel and physical presence in the hospital. COVID applications have shown how AI models can be brittle and not generalize ⁶⁴.

Although some regulatory obstacles were removed, at least temporarily, COVID has not impacted the availability of algorithms that solve real clinical problems. Researchers have moved away from simple segmentation toward clinically useful predictions—such as whether a patient will require hospital admission, will require artificial ventilation, or will have a higher risk of death—but clinicians have not adopted the AI assessments into clinical care with the same enthusiasm.

What Are AI’s “Low-hanging Fruit” for Radiology?

AI has a huge potential to transform medical imaging in various ways by integrating imaging and clinical data in relational databases that can be used for both clinical practice and research, not just for billing ⁶⁵^,⁶⁶. It is important to seamlessly link Digital Imaging and Communications in Medicine (ie, DICOM) data with Fast Healthcare Interoperability Resources (ie, FHIR) data from the EHRs within PACS ⁶⁷^,⁶⁸. Additionally, data scientists and clinicians need to apply the existing well-validated AI tools, such as organ and lesion volumetrics, to the clinical setting. Examples include liver, spleen, and kidney volumetry, visceral fat assessment, and muscle analysis for sarcopenia and myosteatosis ⁶⁹. Finally, there is need to personalize the treatment options for each patient with cancer by using AI analytics for cancer radiomics and longitudinal tracking of individual lesions ⁷⁰^,⁷¹.

What Is Next in AI for Radiology?

The next frontier for medical imaging AI is implementation in clinical practice ⁷². Collaboration is essential among radiologists, referring clinicians, data scientists, and health IT professionals to identify and evaluate AI solutions for various clinical scenarios. New metrics would be useful to measure AI’s impact on workflow efficiency, assess clinical outcomes in radiology and related fields, and conduct randomized clinical trials to assess the utility of AI algorithms ⁵⁵. Promising research directions include unsupervised and self-supervised learning techniques to develop generalizable models that can be adapted to small or niche datasets or to any hospital dataset ⁷³^–⁷⁵. Researchers are also creating broader AI models like foundation models for tasks such as normal versus abnormal detection, rather than narrow supervised ML tasks ⁷⁶^–⁷⁸. Particularly, large vision language models promise new and general-purpose AI applicability in diagnostic imaging through multimodal data analysis to provide a natural language interface for radiologists ⁷⁹. Through these developments, AI can play a crucial role in reducing burnout in radiology if the integration and adoption of AI-driven technologies is done through adaptive and user-friendly tools ⁸⁰.

How Do We Teach and Incentivize Radiologists to Work with AI?

The integration of AI into radiology practice requires educating and motivating radiologists to work with AI ⁸¹. AI must be easy to use within familiar software and compatible with the clinical workflow. Radiologists should learn how to evaluate, interpret, and apply AI algorithms for tasks such as segmentation, detection, and classification, as well as how to handle and report imperfect algorithms ⁸². Radiologists also should learn how to design and conduct clinical trials to assess the impact of AI on clinical outcomes; such training is often lacking in radiology compared with other fields such as oncology. The RSNA’s Clinical Trials Methodology Workshop teaches radiologists about clinical trial design, funding, and implementation. Radiologists also should learn how to communicate and collaborate with AI scientists without feeling intimidated ⁸³. Most radiologists in training are already interested in AI, but they lack a good training mechanism and curriculum. Dedicated training in informatics and involvement in societies such as MICCAI and the Society for Imaging Informatics in Medicine will increase radiologists’ knowledge of AI.

There is also a technical gap between physicians and AI scientists that needs to be bridged ⁸⁴. One way to facilitate the adoption of AI is to use model cards, which describe an AI model’s purpose, usage, and limitations ⁸⁵. Another way is to solve real clinical problems, expand the possibilities of data analyses and interpretation, demonstrate value in prospective trials, and improve workflow efficiency, which would make radiologists eager to use AI without needing additional incentives. A third way is for payers to provide financial incentives to use AI, which would encourage hospitals to invest in AI-based software and increase the motivation for radiologists ⁸⁶.

Unified AI Agenda across Societies and Disciplines

The advancement of AI in medical imaging depends on collaboration across societies and disciplines, which poses both challenges and opportunities. To foster collaboration, it is important to be inclusive, to think beyond personal needs, and to adopt a humanistic and scientific approach to life. Medical imaging societies have collaborated to establish ethical frameworks for AI deployment in radiology and educate clinicians about the potential benefits and pitfalls of AI implementations ⁵⁵. However, differing agendas, languages, and expectations among societies and medical specialties may hinder communication and limit cooperation ¹¹. Therefore, it is essential to create platforms and mechanisms that facilitate the exchange of ideas and information, such as educational programs, challenges and competitions, and hackathons offered by several societies, including joint initiatives of MICCAI and RSNA. Moreover, it is necessary to educate and motivate both technical and radiology societies to understand the capabilities and limitations of AI, the true clinical needs and problems, and the technical skills and methods required to develop and deploy AI solutions. Finally, it is desirable to pursue innovation and discovery by creating and evaluating novel algorithms, integrating and analyzing imaging and clinical data, and developing and applying AI tools for various tasks and domains. By collaborating across societies and disciplines, AI in medical imaging can achieve its full potential and provide value for clinical practice and research.

References

1. Mongan J, Kalpathy-Cramer J, Flanders A, Linguraru MG. RSNA-MICCAI panel discussion: Machine learning for radiology from challenges to clinical applications. Radiol Artif Intell 2021;3(5):e210118.

2. Linguraru MG, Maier-Hein L, Summers RM, Kahn CE Jr. RSNA-MICCAI panel discussion: 2. Leveraging the full potential of ai-radiologists and data scientists working together. Radiol Artif Intell 2021;3(6):e210248.

3. Venkatesh K, Santomartino SM, Sulam J, Yi PH. Code and data sharing practices in the radiology artificial intelligence literature: A meta-research study. Radiol Artif Intell 2022;4(5):e220081.

4. Batlle JC, Dreyer K, Allen B, et al. Data sharing of imaging in an evolving health care world: Report of the ACR Data Sharing Workgroup, Part 1: Data ethics of privacy, consent, and anonymization. J Am Coll Radiol 2021;18(12):1646–1654.

5. Batlle JC, Dreyer K, Allen B, et al. Data sharing of imaging in an evolving health care world: Report of the ACR Data Sharing Workgroup, Part 2: Annotation, curation, and contracting. J Am Coll Radiol 2021;18(12):1655–1665.

6. Rehman MHU, Hugo Lopez Pinaya W, Nachev P, Teo JT, Ourselin S, Cardoso MJ. Federated learning for medical imaging radiology. Br J Radiol 2023;96(1150):20220890.

7. Kwak L, Bai H. The role of federated learning models in medical imaging. Radiol Artif Intell 2023;5(3):e230136.

8. Patel M, Dayan I, Fishman EK, et al. Accelerating artificial intelligence: How federated learning can protect privacy, facilitate collaboration, and improve outcomes. Health Informatics J 2023;29(4):14604582231207744.

9. Darzidehkalani E, Ghasemi-Rad M, van Ooijen PMA. Federated learning in medical imaging: Part ii: Methods, challenges, and considerations. J Am Coll Radiol 2022;19(8):975–982.

10. Willemink MJ, Koszek WA, Hardell C, et al. Preparing medical imaging data for machine learning. Radiology 2020;295(1):4–15.

11. Prevedello LM, Halabi SS, Shih G, et al. Challenges related to artificial intelligence research in medical imaging and the importance of image analysis competitions. Radiol Artif Intell 2019;1(1):e180031.

12. Colak E, Kitamura FC, Hobbs SB, et al. The RSNA Pulmonary Embolism CT Dataset. Radiol Artif Intell 2021;3(2):e200254.

13. Flanders AE, Prevedello LM, Shih G, et al. Construction of a machine learning dataset through collaboration: The RSNA 2019 brain CT hemorrhage challenge. Radiol Artif Intell 2020;2(3):e190211.

14. Shih G, Wu CC, Halabi SS, et al. Augmenting the National Institutes of Health chest radiograph dataset with expert annotations of possible pneumonia. Radiol Artif Intell 2019;1(1):e180041.

15. Demirer M, Candemir S, Bigelow MT, et al. A user interface for optimizing radiologist engagement in image data curation for artificial intelligence. Radiol Artif Intell 2019;1(6):e180095.

16. Galbusera F, Cina A. Image annotation and curation in radiology: an overview for machine learning practitioners. Eur Radiol Exp 2024;8(1):11.

17. Faghani S, Khosravi B, Zhang K, et al. Mitigating bias in radiology machine learning: 3. Performance metrics. Radiol Artif Intell 2022;4(5):e220061.

18. Rouzrokh P, Khosravi B, Faghani S, et al. Mitigating bias in radiology machine learning: 1. Data handling. Radiol Artif Intell 2022;4(5):e210290.

19. Zhang K, Khosravi B, Vahdati S, et al. Mitigating bias in radiology machine learning: 2. Model development. Radiol Artif Intell 2022;4(5):e220010.

20. Stytz MR, Frieder G, Frieder O. Three-dimensional medical imaging: algorithms and computer systems. ACM Comput Surv 1991;23(4):421–499.

21. Batchelor PG, Edwards PJ, King AP. 3D Medical Imaging. In: Pears N, Liu Y, Bunting P, eds. 3D Imaging, Analysis and Applications. Springer, 2012; 445–495.

22. Phelps EE, Wellings R, Griffiths F, Hutchinson C, Kunar M. Do medical images aid understanding and recall of medical information? An experimental study comparing the experience of viewing no image, a 2D medical image and a 3D medical image alongside a diagnosis. Patient Educ Couns 2017;100(6):1120–1127.

23. Hopper KD, Kasales CJ, Eggli KD, et al. The impact of 2D versus 3D quantitation of tumor bulk determination on current methods of assessing response to treatment. J Comput Assist Tomogr 1996;20(6):930–937.

24. Chapiro J, Lin M, Duran R, Schernthaner RE, Geschwind JF. Assessing tumor response after loco-regional liver cancer therapies: the role of 3D MRI. Expert Rev Anticancer Ther 2015;15(2):199–205.

25. Aghighi M, Boe J, Rosenberg J, et al. Three-dimensional radiologic assessment of chemotherapy response in ewing sarcoma can be used to predict clinical outcome. Radiology 2016;280(3):905–915.

26. Hausmann D, Liu J, Budjan J, et al. Image Quality Assessment of 2D versus 3D T2WI and Evaluation of Ultra-high b-Value (b=2,000 mm/s2) DWI for Response Assessment in Rectal Cancer. Anticancer Res 2018;38(2):969–978.

27. Devaraj A, van Ginneken B, Nair A, Baldwin D. Use of volumetry for lung nodule management: Theory and practice. Radiology 2017;284(3):630–644.

28. Davatzikos C, Rathore S, Bakas S, et al. Cancer imaging phenomics toolkit: quantitative imaging analytics for precision diagnostics and predictive modeling of clinical outcome. J Med Imaging (Bellingham) 2018;5(1):011018.

29. Yousefirizi F, Pierre Decazes, Amyar A, Ruan S, Saboury B, Rahmim A. AI-Based Detection, Classification and Prediction/Prognosis in Medical Imaging: Towards Radiophenomics. PET Clin 2022;17(1):183–212.

30. Pickhardt PJ, Graffy PM, Zea R, et al. Automated CT biomarkers for opportunistic prediction of future cardiovascular events and mortality in an asymptomatic screening population: a retrospective cohort study. Lancet Digit Health 2020;2(4):e192–e200.

31. Goodacre S. Using clinical risk models to predict outcomes: what are we predicting and why? Emerg Med J 2023;40(10):728–730.

32. Saunders MB, Gulabivala K, Holt R, Kahan RS. Reliability of radiographic observations recorded on a proforma measured using inter- and intra-observer variation: a preliminary study. Int Endod J 2000;33(3):272–278.

33. Gasperini C, Rovaris M, Sormani MP, et al. Intra-observer, inter-observer and inter-scanner variations in brain MRI volume measurements in multiple sclerosis. Mult Scler 2001;7(1):27–31.

34. Alikhassi A, Esmaili Gourabi H, Baikpour M. Comparison of inter- and intra-observer variability of breast density assessments using the fourth and fifth editions of Breast Imaging Reporting and Data System. Eur J Radiol Open 2018;5:67–72.

35. Pakdemirli E. Perception of Artificial Intelligence (AI) among radiologists. Acta Radiol Open 2019;8(9):2058460119878662.

36. Schuur F, Rezazade Mehrizi MH, Ranschaert E. Training opportunities of artificial intelligence (AI) in radiology: a systematic review. Eur Radiol 2021;31(8):6021–6029.

37. Hu R, Rizwan A, Hu Z, Li T, Chung AD, Kwan BYM. An artificial intelligence training workshop for diagnostic radiology residents. Radiol Artif Intell 2023;5(2):e220170.

38. van Kooten MJ, Tan CO, Hofmeijer EIS, et al. A framework to integrate artificial intelligence training into radiology residency programs: preparing the future radiologist. Insights Imaging 2024;15(1):15.

39. Dikici E, Bigelow M, Prevedello LM, White RD, Erdal BS. Integrating AI into radiology workflow: levels of research, production, and feedback maturity. J Med Imaging (Bellingham) 2020;7(1):016502.

40. Ranschaert E, Topff L, Pianykh O. Optimization of radiology workflow with artificial intelligence. Radiol Clin North Am 2021;59(6):955–966.

41. Mongan J, Avrin D. Impact of PACS-EMR integration on radiologist usage of the EMR. J Digit Imaging 2018;31(5):611–614.

42. Bergquist M, Rolandsson B, Gryska E, et al. Trust and stakeholder perspectives on the implementation of AI tools in clinical radiology. Eur Radiol 2024;34(1):338–347.

43. Lim SS, Phan TD, Law M, et al. Non-radiologist perception of the use of artificial intelligence (AI) in diagnostic medical imaging reports. J Med Imaging Radiat Oncol 2022;66(8):1029–1034.

44. Hadjiiski L, Cha K, Chan HP, et al. AAPM task group report 273: Recommendations on best practices for AI and machine learning for computer-aided diagnosis in medical imaging. Med Phys 2023;50(2):e1–e24.

45. Lekadir K, Feragen A, Fofanah AJ, et al. FUTURE-AI: International consensus guideline for trustworthy and deployable artificial intelligence in healthcare. arXiv 2309.12325 [preprint] https://arxiv.org/abs/2309.12325 . Posted August 11, 2023. Accessed June 2024.

46. Omoumi P, Ducarouge A, Tournier A, et al. To buy or not to buy-evaluating commercial AI solutions in radiology (the ECLAIR guidelines). Eur Radiol 2021;31(6):3786–3796.

47. Anazodo UC, Adewole M, Dako F. AI for population and global health in radiology. Radiol Artif Intell 2022;4(4):e220107.

48. Abuzaid MM, Elshami W, Tekin H, Issa B. Assessment of the willingness of radiologists and radiographers to accept the integration of artificial intelligence into radiology practice. Acad Radiol 2022;29(1):87–94.

49. Martín-Noguerol T, Paulano-Godino F, López-Ortega R, Górriz JM, Riascos RF, Luna A. Artificial intelligence in radiology: relevance of collaborative work between radiologists and engineers for building a multidisciplinary team. Clin Radiol 2021;76(5):317–324.

50. Alexander R, Waite S, Bruno MA, et al. Mandating limits on workload, duty, and speed in radiology. Radiology 2022;304(2):274–282.

51. Tanguay W, Acar P, Fine B, et al. Assessment of radiology artificial intelligence software: A validation and evaluation framework. Can Assoc Radiol J 2023;74(2):326–333.

52. Allen B, Dreyer K, Stibolt R Jr, et al. Evaluation and real-world performance monitoring of artificial intelligence models in clinical practice: Try it, buy it, check it. J Am Coll Radiol 2021;18(11):1489–1496.

53. Soltaninejad M, Yang G, Lambrou T, et al. Supervised learning based multimodal MRI brain tumour segmentation using texture features from supervoxels. Comput Methods Programs Biomed 2018;157:69–84.

54. Pianykh OS, Langs G, Dewey M, et al. Continuous learning AI in radiology: Implementation principles and early applications. Radiology 2020;297(1):6–14.

55. Brady AP, Allen B, Chong J, et al. Developing, purchasing, implementing and monitoring AI tools in radiology: practical considerations. A multi-society statement from the ACR, CAR, ESR, RANZCR and RSNA. Radiol Artif Intell 2024;6(1):e230513.

56. Harvey HB, Gowda V. How the FDA Regulates AI. Acad Radiol 2020;27(1):58–61.

57. Artificial Intelligence and Machine Learning (AI/ML)-Enabled Medical Devices. U.S. Food and Drug Administration . https://www.fda.gov/medical-devices/software-medical-device-samd/artificial-intelligence-and-machine-learning-aiml-enabled-medical-devices . Published 2023. Updated October 19, 2023. Accessed March 2024.

58. Khunte M, Chae A, Wang R, et al. Trends in clinical validation and usage of US Food and Drug Administration-cleared artificial intelligence algorithms for medical imaging. Clin Radiol 2023;78(2):123–129.

59. Benjamens S, Dhunnoo P, Meskó B. The state of artificial intelligence-based FDA-approved medical devices and algorithms: an online database. NPJ Digit Med 2020;3(1):118.

60. Li J, Long X, Wang X, et al. Radiology indispensable for tracking COVID-19. Diagn Interv Imaging 2021;102(2):69–75.

61. Bai HX, Wang R, Xiong Z, et al. Artificial intelligence augmentation of radiologist performance in distinguishing COVID-19 from pneumonia of other origin at chest CT. Radiology 2020;296(3):E156–E165.

62. Bouchareb Y, Moradi Khaniabadi P, Al Kindi F, et al. Artificial intelligence-driven assessment of radiological images for COVID-19. Comput Biol Med 2021;136:104665.

63. Medical Imaging and Data Resource Center. https://www.midrc.org/ . Accessed March 2024.

64. Topff L, Groot Lipman KBW, Guffens F, et al. Is the generalizability of a developed artificial intelligence algorithm for COVID-19 on chest CT sufficient for clinical use? Results from the International Consortium for COVID-19 Imaging AI (ICOVAI). Eur Radiol 2023;33(6):4249–4258.

65. Phan JH, Quo CF, Cheng C, Wang MD. Multiscale integration of -omic, imaging, and clinical data in biomedical informatics. IEEE Rev Biomed Eng 2012;5:74–87.

66. Holzinger A, Haibe-Kains B, Jurisica I. Why imaging data alone is not enough: AI-based integration of imaging, omics, and clinical data. Eur J Nucl Med Mol Imaging 2019;46(13):2722–2730.

67. Prasser F, Kohlbacher O, Mansmann U, Bauer B, Kuhn KA. Data Integration for Future Medicine (DIFUTURE). Methods Inf Med 2018;57(S 01):e57–e65.

68. Kaspar M, Liman L, Morbach C, et al. Querying a clinical data warehouse for combinations of clinical and imaging data. J Digit Imaging 2023;36(2):715–724.

69. Pickhardt PJ, Summers RM, Garrett JW, et al. Opportunistic screening: Radiology scientific expert panel. Radiology 2023;307(5):e222044.

70. Meng Y, Sun J, Qu N, Zhang G, Yu T, Piao H. Application of radiomics for personalized treatment of cancer patients. Cancer Manag Res 2019;11:10851–10858.

71. Cassinelli Petersen G, Bousabarah K, Verma T, et al. Real-time PACS-integrated longitudinal brain metastasis tracking tool provides comprehensive assessment of treatment response to radiosurgery. Neurooncol Adv 2022;4:vdac116.

72. Dreyer KJ, Geis JR. When machines think: Radiology’s next frontier. Radiology 2017;285(3):713–718.

73. Bernhardt M, Castro DC, Tanno R, et al. Active label cleaning for improved dataset quality under resource constraints. Nat Commun 2022;13(1):1161.

74. Wu S, Miller T, Masanz J, et al. Negation’s not solved: generalizability versus optimizability in clinical natural language processing. PLoS One 2014;9(11):e112774.

75. Huang SC, Pareek A, Jensen M, Lungren MP, Yeung S, Chaudhari AS. Self-supervised learning for medical image classification: a systematic review and implementation guidelines. NPJ Digit Med 2023;6(1):74.

76. Eyuboglu S, Angus G, Patel BN, et al. Multi-task weak supervision enables anatomically-resolved abnormality detection in whole-body FDG-PET/CT. Nat Commun 2021;12(1):1880.

77. Moor M, Banerjee O, Abad ZSH, et al. Foundation models for generalist medical artificial intelligence. Nature 2023;616(7956):259–265.

78. Anwar SM, Parida A, Atito S, et al. SPCXR: Self-supervised pretraining using chest x-rays towards a domain specific foundation model. arXiv 2211.12944 [preprint] https://arxiv.org/abs/2211.12944 . Posted November 23, 2022. Accessed June 2024.

79. Li C, Wong C, Zhang S, et al. LLaVA-Med: Training a Large Language-and-Vision Assistant for Biomedicine in One Day. In: Advances in Neural Information Processing Systems 36 (NeurIPS 2023) Datasets and Benchmarks Track . 2023. https://proceedings.nips.cc/paper_files/paper/2023/hash/5abcdf8ecdcacba028c6662789194572-Abstract-Datasets_and_Benchmarks.html .

80. Wenderott K, Krups J, Luetkens JA, Weigl M. Radiologists’ perspectives on the workflow integration of an artificial intelligence-based computer-aided detection system: A qualitative study. Appl Ergon 2024;117:104243.

81. Tejani AS, Elhalawani H, Moy L, Kohli M, Kahn CE Jr. Artificial intelligence and radiology education. Radiol Artif Intell 2022;5(1):e220084.

82. Wiggins WF, Caton MT, Magudia K, et al. Preparing radiologists to lead in the era of artificial intelligence: Designing and implementing a focused data science pathway for senior radiology residents. Radiol Artif Intell 2020;2(6):e200057.

83. Dewey M, Bosserdt M, Dodd JD, Thun S, Kressel HY. Clinical imaging research: Higher evidence, global collaboration, improved reporting, and data sharing are the grand challenges. Radiology 2019;291(3):547–552.

84. Weggemans MM, Friesen F, Kluijtmans M, et al. Critical gaps in understanding the clinician-scientist workforce: Results of an international expert meeting. Acad Med 2019;94(10):1448–1454.

85. Mitchell M, Wu S, Zaldivar A, et al. Model cards for model reporting. In: FAT* ‘19: Proceedings of the Conference on Fairness, Accountability, and Transparency. https://doi.org/10.1145/3287560.3287596 . Published January 29, 2019. Accessed August 2022.

86. Ashikyan O, Xia S, Faridi O, Porembka JH, Chhabra A. Positive effect of a financial incentive on radiologist compliance with quality metric placement in knee radiography reports. J Am Coll Radiol 2024. Published online January 30, 2024.