Open Source AI Model for COVID-19 CT Diagnostics and Prognosis Summary – Experiment

Vancouver General Hospital radiologists began looking into the feasibility of developing an AI model in January 2020, focusing on analyzing CT scans of COVID-19 infected patients. The goal became to empower radiologists and provide metrics and statistical information about the infection that can’t normally be assessed by the human eye alone.

At that time, 3 in 10 people who had the disease were testing negative and sent home due to the massive overcrowding of hospitals and lack of resources to take care of them. This impacted infection rates and there was a palpable demand for additional tools or tests to be developed to help provide supplemental information to augment a physician’s decision-making toolbox.

In the initial phase of work, an ensemble cast was assembled to help foster the growth of a collaborative community that involved Vancouver General Hospital, Vancouver Imaging, SapienML, Amazon Web Services, Xtract.ai, the University of British Columbia, Element ai, and md.ai. This collaboration allowed us to reach out to healthcare centers from around the world and collect CT scan data from USA, China, South Korea, Australia, Canada, Italy and the Middle East of patients that ranged in diversity and infection severity.

This first phase resulted in Version 1 of L3-Net and enabled the new, improved, scalable L3-Net model of today.

Timeline

• January 2023
   

  Phase 3 Publication

   

• JAN 01, 2020
   

  2023

  Description Placeholder

• October 2022
   

  Installed Model at Vancouver General Hospital

   

• July 2022
   

  Phase 2 Model Publication

   

• June 2022
   

  Model Training

   

• Jan 2022
   

  Annotations of Phase 3 Data

   

• JAN 01, 2020
   

  2022

  Description Placeholder

• August 2021
   

  Phase 3 Planning and Development Road Map

   

• June 2021
   

  Data Analysis

   

• February 2021
   

  Version 2.0 Model Release

  Prognostication model version 2.0 release.

• JAN 01, 2020
   

  2021

  Description Placeholder

• August 2020
   

  Complete Final Phase 3 Model

   

• August 2020
   

  Version 1.0 Model Release

  Official release of version 1.0 of the AI model and associated tools.

• July 2020
   

  Beta Prototype

  The Beta Prototype v0.9 release with free app.

• June 2020
   

  Alpha Prototype

  The Alpha prototype v0.1 release on Github.

The model was built to be open-source, so that centres around the world could continue to develop on their own patient data to further improve the performance of the model, as well as come back and share their findings with the community. Fostering an open-source community and mentality around treating the illness may be the fastest and safest way to cross the patient-privacy barrier and foster enhanced collaboration between sites around the world.

If your team wishes to label your data to prepare for training on top of L3-Net, please do not hesitate to reach out to our team: hello@sapienml.com

Data

Just like quality ingredients are the key to a delicious meal, quality data is the building block of powerful AI. SapienSecure prepared a dataset that is comprised of COVID-19 positive scans as well as scans of patients that present with similar symptoms but had a different type of pneumonia.

Fostering an open-source community and mentality around treating this illness may be the fastest and safest way to… foster enhanced collaboration between sites arounds the world.

Brian Lee, SapienSecure CTO

The dataset contains CT studies from all around the globe, including USA, China, South Korea, Australia, Canada, Italy and the Middle East. This was done to increase model generalizability, minimize bias and establish an accurate model for any site to either pull off the shelf and use, or reinforce with data from their own local population. As the goal of this project was to foster a collaborative development and research community, it was important that the dataset was not highly specific to any one region. The dataset also contained images from scanners made by many different manufacturers, including ones that are not common to the sites local to the developers. Not only does scanner/image quality vary, but so does the collection techniques and image post-processing methods. The scans vary in thickness as well as reconstruction techniques, which our model tachles flexibly.

Labels

SapienSecure lives and breathes medicine, and so when it comes to labeling high quality datasets, we turn to our medical experts. Dozens of radiologists, residents, fellows and medical students were brought onboard from Vancouver, Canada to create this model.

These Radiologists provide expert opinions on the presence and location of lung abnormalities caused by pneumonia (including COVID-19), as well as analyze the severity of the infection. As multiple radiologists gave their opinions, we aimed to eliminate bias in our annotations, as well as decrease variability by a radiologist’s interpretation of a scan.

Annotations were broken into three broad categories:

1. Normal Lung: This model output annotates the normal lung parenchyma, informing the model user of the proportion of lung NOT involved with the infection. This is kind of analogous to the pulmonary reserve that the patient has left in order to overcome any infectious process.
2. Ground Glass Opacity: Ground glass is a sign of inflammation, and tends to represent a milder form of inflammation than consolidation. It is a non-specific lung abnormality, but is abnormal none-the-less. The model outputs the proportion of lung that contains ground glass (GGO).
3. Consolidation: Consolidation is a sign of inflammation, and tends to represent a more severe form when compared to ground glass opacities. It is also a non-specific lung abnormality, and is a common defining characteristic of pneumonia. The model outputs the proportion of lung that contains consolidation.

L3-net

L3-net is not just another CT AI model. We wanted to empower medical personnel with tools that would augment their ability to make decisions involving patients with severe pneumonia ( such as COVID-19). L3-net is a clinically relevant approach to AI in medicine and is a continually evolving project that aims to provide quantitative measurements to radiologists. The team behind L3-net has worked closely with Medical Doctors called Radiologists to identify the features and characteristics in CT scans that correlate most strongly with poor outcomes in patients. Every facet of the model has grown organically from the needs of the radiologists involved with this project. Daily interaction between the Engineering team and Medical staff has shaped the development of L3-net every step of the way.

L3-Net performs well. The following are preliminary results:

We have worked with health centers around the world to put together one of the largest international CT chest datasets.

The World Health Organization declared the 2019-20 coronavirus outbreak a Public Health Emergency.

W.H.O.

One of the key elements of this project is that we are aiming to empower doctors with more information than they ever had before. The model provides analytics that help a Doctor make critical decisions.

AI is no good if it’s not accessible. One of our core tenets is to release everything open source. The architecture, the training algorithms, and even the weights were made publicly available to download. We promote discussion and criticism to foster a community that comes together to fight back against CoVID-19 and other infections.

Proudly Canadian. 🇨🇦

Explore The Tool And The Code

The amazing Team that worked on the project are listed here. The models and code are available on Github. We encourage you to explore both. If you have questions about our approach, code, and/or labels, please reach out!  hello@sapienml.com