Building an intelligent diagnosis system for the real-world clinical applications using chest X-ray images

There are currently limited examples of intelligent diagnostic models being successfully deployed into real-world clinical applications, due to underlying dataset biases, methodological flaws or non-peer-reviewed clinical trials.
Building an intelligent diagnosis system for the real-world clinical applications using chest X-ray images

Chest X-ray (CXR) radiography is the frontline tool and mainstay of screening, triaging, and diagnosing varieties of pneumonia, including bacterial, viral, and other types of pneumonia, as well as common chest disorders. 

Key challenges for delivering clinical impact with artificial intelligence

Recent developments in artificial intelligence (AI) have provided new potential opportunities for the rapid growth of radiological diagnostic applications. Many articles have been published during the COVID-19 pandemic, describing AI-based models for fast and accurate detection and prognostication of disease using chest radiographs (CXR) or chest computed tomography (CT) images.

However, there are currently limited examples of intelligent diagnostic models being successfully deployed into real-world clinical applications, due to underlying dataset biases, methodological flaws or non-peer-reviewed clinical trials [1]. For example, researcher usually assemble the datasets by using a variety of methods, from the manual gathering of samples up to crawlers to parse through the Internet and other publicly available repositories. However, one drawback of many public datasets is that it is not possible to know whether patients were truly COVID-19 positive by a confirmed molecular test. Additionally, some papers have serious flaws in that they were using viral and bacterial pneumonia images from pediatric populations and comparing them to COVID-19 CXR images from an adult/elderly population in training their AI systems, which seriously bias the AI system in learning differences in demographics, not inherent disease features.

To address these challenges, this work has been performed by both algorithm developers and clinicians. Additionally, we explored several key factors, including: real-world dataset construction, a general AI system development, and clinical trials with external cohorts from different populations [2]. 

An automated deep learning pipeline for common lung diseases

Here we developed a comprehensive AI system to detect chest diseases, combat the COVID-19 or any other emerging upper respiratory viral pandemic.  A plain CXR image is the summation of the effect of X-ray on all tissues between the X-ray source and the capturing film; tissue structures are less well defined in an X-ray image comparing to a CT image and lack 3-dimensional information. To overcome these shortcomings, we integrate multiple state-of-the-art computational methods to construct a robust AI system for CXR diagnosis.  Technically, our AI system is a modular analysis pipeline consisting of three modules: a CXR standardization module, a common thoracic disease detection module, and a final pneumonia analysis module.

Figure 1. Schematic illustration of our proposed AI platform for common chest diseases detection and and pneumonia analysis.

The CXR standardization module consisted of anatomical landmarks detection and image registration techniques (Fig. 1). This module was designed to overcome the notorious problem and well-known challenges of data diversity/variations and non-standardization of CXR images. The common thoracic disease detection module classified the standardized CXR images into 14 common thoracic pathologies that are frequently observed and diagnosed, including cardiomegaly, consolidation, edema, effusion, emphysema, fibrosis, hernia, infiltration, mass, nodule, pleural thickening, pneumonia, and pneumothorax. The pneumonia analysis module that consists of a lung-lesion segmentation model and a final classification model estimates the subtype of pneumonia (e.g., viral pneumonia) and assesses the severity of COVID-19.

CC-CXRI: a large chest radiograph dataset from real-world 

The AI methods  depend on the quality and reliability of the labeled datasets. Here we present a large-scale, heterogeneous, multi-centre dataset based on the China Consortium of Chest X-ray Image Investigation (CC-CXRI). The CC-CXRI consisted of two large-scale datasets: the first, a CXR database for common thoracic diseases containing retrospectively 145,202 CXR images, and the second, a CXR dataset (CC-CXRI-P) containing 16,196 CXR images for detecting suspicious pneumonia, including COVID-19 pneumonia. Compared with the existing open-source CXR datasets, CC-CXRI includes relevant variations in patient demographics and disease states of target patients in real-world clinical settings, including inpatient, outpatient, or patient for physical examination. For example, the outpatient department contains patients with various mild lung abnormalities, which can be used to model healthy people and build an AI method for abnormal disease detection.

Both the source code and the annotated dataset have been made public to assist other investigators using our systems as our contribution to help the world to combat disease and to benefit future research in this direction (

The performance of this general AI system

 This study showed a few crucial points. First, a noteworthy feature of the AI system is that the modular processing pipeline, including modules of anatomical landmark detection, registration, lung-lesion segmentation, and diagnosis prediction, provided robust and explainable results. Despite the limitation of plain CXR, this accurate AI system can be established to assist radiologists to accurately identify viral pneumonia as well as COVID-19, which can be very useful as a frontline tool in an emergency clinic, remote places, or the developing world. Second, we compared the performance of the system with that of radiologists in routine clinical practice. This AI system can help junior radiologists to perform close to the level of senior radiologists. Finally, this system can differentiate COVID-19 from other types of viral pneumonia with reasonable accuracy. The AI system can also accurately determine the severity of the lesions in patients with established COVID-19. Overall, this diagnostic tool can assist radiologists in many clinical settings, from facilitating early intervention and clinical decision support to the overall assessment of severity when chest CT imaging is not readily available.

 Although there are published studies on AI-based diagnosis of pneumonias, the actual clinical applicability remains unknown since they have not been shown to be free of experimental data bias, and they have not been tested by the peer-reviewed gold standard labels and by external data in different populations, or in different clinical settings to show generalizability. In this paper, we explored the general applicability of the current AI system. We first trained our AI system using large, heterogenous, multi-center datasets. Then we present evidence of the ability of AI systems to translate between different populations/settings. For example, we trained the model for common thoracic diseases on patients for hospital visits, and then measured performance on populations for physical examination with populations of less chest pathology compared with the training set. In this context, the system continued to achieve accurate performance. This practice is rare in current literature and is of critical concern.

What’s next?

We’ll continue to focus on the main challenges and limitations of AI in healthcare, and the steps required to translate these potentially transformative technologies from research to clinical practice. Further work to improve the interpretability of algorithms and to understand human–algorithm interactions will be essential to their future adoption to a real-world scenario.


  1. Roberts, M., Driggs, D., Thorpe, M. et al. Common pitfalls and recommendations for using machine learning to detect and prognosticate for COVID-19 using chest radiographs and CT scans. Nature Machine Intelligence 3, 199–217 (2021).
  2. Guangyu Wang*#, Xiaohong Liu*, Jun Shen*, et al. Kang Zhang#, Weimin Li#, Tianxin Lin#. (2021). A deeplearning pipeline for the diagnosis and discrimination of viral, non-viral and COVID-19 pneumonia from chest X-ray images. Nature Biomedical Engineering,