Phoebe Qian and James Setty: “Communication Methods and Workflows to Develop and Implement High Throughput Truthing of Pathologist Annotations as a Reference Standard for Validating Artificial Intelligence in Digital Pathology”
Abstract: Due to a lack of standard reference data, it is a challenge to validate artificial intelligence (AI) algorithms for clinical use in medical images. Many research papers in AI are focused on developing novel algorithms and only moderately address how to accurately evaluate performance. The High-Throughput Truthing Project (HTT) aims to crowdsource pathologist annotations on digital pathology images to produce an algorithm validation data set. We will pursue qualification of the images and annotations by the FDA/CDRH medical device development tool program (MDDT). A pilot study using an optical microscope and digital platforms provided promising results but also revealed multiple improvements that could be made to the data-collection methods. The current workflow for the project can lead to participants incorrectly completing or missing key steps of the data-collection process, decreasing the overall quality of the resulting data. Study size, patient population, pathologist training and pathologist qualifications are key considerations when creating the data set for regulatory purposes and will be emphasized when making improvements to the data collection-process. In this poster, we outline the evaluation and improvements of the data-collection process. Changes are made with the intent to decrease the probability of participants taking actions during data collection that will degrade the overall quality of the collected data. We make these alterations with the goal of creating publicly available statistical methods and a dataset that can be used by the broader digital pathology and AI community for validating algorithms.
Title: Communication Methods and Workflows to Develop and Implement High Throughput Truthing of Pathologist Annotations as a Reference Standard for Validating Artificial Intelligence in Digital Pathology.
Authors: Phoebe Qian, James Setty, Katherine Elfer, Brandon Gallas
Abstract: Artificial intelligence (AI) describes machines or computers that mimic “cognitive” functions associated with the human mind, such as “learning” and “problem-solving”. Recent advances in algorithm architecture, software tools, hardware infrastructure, and regulatory frameworks have enabled healthcare stakeholders to harness AI as a medical device. Such AI medical devices promise to reduce pathologists’ burden in searching and evaluating cells and features on the slides. Algorithms/software should be validated as a medical device before it can be marketed and applied in the clinical workflow to ensure safety and effectiveness before clinical deployment. In the High-throughput Truthing (HTT) project, we present a collaboration to create a validation dataset of pathologist annotations for algorithms that process whole slide images (WSIs). Validating AI algorithms for clinical use in medical images is a challenging endeavor due to a lack of standard reference data (ground truth). As the first systematic project to collect standard pathologist annotations from various platforms, this project has stakeholders from a variety of statisticians, researchers, and developers to analyze our data and develop data-collecting technologies, and pathologists to train and annotate slides. Not only do we have to adapt our communication methods to a broad audience, but we also have to get technical with their specific areas of interest. We accommodate our communication needs by utilizing the tools of various online resources including, R package documentation, NCI Hub groups, and R-script files that assist in diving deeper into analyzing data. This poster will include visual descriptions of standardizing our communication platforms to improve our workflows for everyone in the project.