Skip to main content

Presentation 2021: Pathology Informatics Summit - HTT project

Back to Presentations and Publications Main Page

Pathology Informatics Summit 2021

Title: High Throughput Truthing (HTT): Pathologist Agreement from a Pilot Study

  • 11:30 AM, Thursday, 06 May 2021
  • Virtual

Final presentation slides

20210506_PIsummit-Gallas-share.pdf (2 MB, uploaded by Brandon D. Gallas 1 year 10 months ago)



  • Artificial intelligence algorithms in digital pathology have enormous potential to increase diagnostic speed and accuracy. However, the performance of these algorithms must be validated against a reference standard before deployment in clinical practice. In this work, pathologists are considered as the reference standard. We studied interobserver variability in pathologists who evaluate stromal tumor-infiltrating lymphocytes (sTILs) in hematoxylin and eosin stained breast cancer biopsy specimens. Our ultimate goal is to create a validation dataset that is fit for a regulatory purpose.


  • Following an IRB exempt determination protocol, we obtained informed consent of volunteer pathologist annotators prior to completing data collection tasks via two modalities: an optical light microscope and two digital platforms (slides were scanned at 40X). Pathologists were trained on the clinical task of sTIL density estimation before annotating pre-specified regions of interest (ROIs) across multiple platforms. The ROI selection protocol sampled ROIs in the tumor, tumor boundary, and elsewhere. Inter-pathologist agreement was characterized with the root mean-squared difference, which is analogous to the root mean-squared error but doesn’t require ground truth.


  • The pilot study accumulated 6,257 sTIL density estimates from 34 pathologists evaluating 64 cases, with 10 ROIs per case. The variability of sTIL density estimates in an ROI increases with the mean; the square roots of the reader-averaged mean-squared differences were 8.3, 17.7, and 40.4 as the sTIL density reference score increased from 0-10%, 10-40%, and 40-100%, respectively. We also found that the root mean-squared differences for some pathologists were considerably larger than others (as much as 120% larger than the next largest root mean-squared difference).


  • Slides, images, and annotations were successfully provided by volunteer collaborators and participants, which created an innovative and thorough method for data collection and truthing. This pilot study will inform the development of statistical methods, simulation models, and sizing analyses for pivotal studies. The development and results of this validation dataset and analysis tools will be made publicly available to serve as an instructive tool for algorithm developers and researchers. Furthermore, the methods used to analyze pathologist agreement between density estimates are applicable to other quantitative biomarkers.


  • Katherine Elfer, PhD, MPH
  • Mohamed Amgad, MD
    • Department of Pathology, Northwestern University
  • Weijie Chen, PhD
  • Sarah Dudgeon, MPH
    • CORE Center for Computational Health Yale-New Haven Hospital
  • Rajarsi Gupta, MD/PhD
    • Stony Brook Medicine Dept of Biomedical Informatics
  • Matthew Hanna, MD
    • Memorial Sloan Kettering Cancer Center
  • Steven Hart, PhD
    • Department of Health Sciences Research, Mayo Clinic
  • Richard Huang, MD
    • Massachusetts General Hospital/Harvard Medical School
  • Evangelos Hytopoulos, PhD
  • Denis Larsimont, MD
  • Xiaoxian Li, MD/PhD
    • Emory University School of Medicine
  • Anant Madabhushi, PhD
    • Case Western Reserve University
  • Hetal Marble, PhD
    • Massachusetts General Hospital/Harvard Medical School
  • Roberto Salgado, PhD
    • Division of Research, Peter Mac CallumCancer Centre, Melbourne, Australia; Department of Pathology, GZA-ZNA Hospitals
  • Joel Saltz, MD/PhD
    • Stony Brook Medicine Dept of Biomedical Informatics
  • Manasi Sheth, PhD
    • FDA/CDRH/OPQE/Division of Biostatistics
  • Rajendra Singh, MD
    • Northwell health and Zucker School of Medicine
  • Evan Szu, PhD
    • Arrive Bio
  • Darick Tong, MS
    • Arrive Bio
  • Si Wen, PhD
  • Bruce Werness, MD
    • Arrive Bio

Created on , Last modified on