Diagnostic agreement between a commercial AI system and radiologists in interpreting checkup chest radiographs

Abstract

Background: Chest radiography is widely used to screen for thoracic diseases. Recent artificial intelligence (AI) systems have demonstrated outstanding standalone performance in diagnostic tasks concerning chest radiographs (CXRs), often comparable to that of radiologists. However, few studies have evaluated AI performance in real-world clinical practice settings, especially with regard to checkup examinations and diagnostic agreement with humans. Objective: To evaluate the diagnostic agreement between AI and radiologists in evaluating checkup CXRs. Methods: An AI system and radiologists independently evaluated 500 checkup CXRs from a retrospective review period. We then quantified their diagnostic agreement. Results: When our analysis was restricted to lesions identified by the AI (AI-target lesions), we found fair agreement. If we regard the diagnoses of the radiologists as ground truth, the AI produced false-negative and false-positive rates of 8.0% and 8.7%, respectively. When extended to include lesions that had not been identified by the AI (both AI-target and AI-non-target lesions), our analysis showed a reduced agreement (76.0%) and an increased false-negative rate (16.7%). The AI demonstrated low sensitivity (26.5–36.8) but high specificity (90.1–90.5), with a significant number of AI-non-target lesions. Conclusions: Our results demonstrate fair to slight agreement between automatic AI diagnoses and the assessments of radiologists for checkup CXRs. Radiologists often reported numerous AI-non-target lesions. Despite certain limitations, our findings suggest that AI may play a valuable role as a diagnostic aid to enhance radiological evaluations. Further research is warranted to comprehensively assess AI performance in broader clinical contexts.