Skip to main content

Presentation 2022 JSM: Exploring Pathologist Agreement

ASA Joint Statistical Meeting Presentation 7 August 2022

  • Presenter: Brandon D Gallas, Ph.D.
  • Title: Exploring Pathologist-Pathologist Agreement as a Baseline for Algorithm-Pathologist Agreement


Gallas2022_JSM-final20220808.pdf (3 MB, uploaded by Brandon D. Gallas 3 months 2 weeks ago)


Expert clinical annotations are a common reference standard for training and testing artificial intelligence and machine learning (AI/ML) models. Depending on the task, these annotations can be fraught with biases and variability. We collected biomarker annotations (density values: 0-100%) from multiple pathologists on specific regions of interest (ROIs) nested within independent cases. After exploring the data, we found the ROIs elicit different levels of reader variability and this variability exhibits a complicated relationship with the mean that cannot be stabilized with a transformation. Hence the data are not independent and identically distributed according to a normal distribution, and the data cannot be pooled and analyzed by standard quantitative analysis methods, like correlation, ANOVA, and regression. This led us to investigate the use of thresholds and ordinal rank-based methods, such as ROC analysis, for assessing pathologist agreement and AI/ML performance. The thresholds allow us to understand agreement at specific biomarker density values (calibration) while the ordinal rank-based methods sidestep distributional assumptions of the data.

Created on , Last modified on