Skip to main content
The NCI Community Hub will be retiring in May 2024. For more information please visit the NCIHub Retirement Page:https://ncihub.cancer.gov/groups/ncihubshutdown/overview
close

Presentation 2022 JSM: Exploring Pathologist Agreement

ASA Joint Statistical Meeting Presentation 7 August 2022

  • Presenter: Brandon D Gallas, Ph.D.
  • Title: Exploring Pathologist-Pathologist Agreement as a Baseline for Algorithm-Pathologist Agreement

Slides

Gallas2022_JSM-final20220808.pdf (3 MB, uploaded by Brandon D. Gallas 1 year 3 months ago)

Abstract

Expert clinical annotations are a common reference standard for training and testing artificial intelligence and machine learning (AI/ML) models. Depending on the task, these annotations can be fraught with biases and variability. We collected biomarker annotations (density values: 0-100%) from multiple pathologists on specific regions of interest (ROIs) nested within independent cases. After exploring the data, we found the ROIs elicit different levels of reader variability and this variability exhibits a complicated relationship with the mean that cannot be stabilized with a transformation. Hence the data are not independent and identically distributed according to a normal distribution, and the data cannot be pooled and analyzed by standard quantitative analysis methods, like correlation, ANOVA, and regression. This led us to investigate the use of thresholds and ordinal rank-based methods, such as ROC analysis, for assessing pathologist agreement and AI/ML performance. The thresholds allow us to understand agreement at specific biomarker density values (calibration) while the ordinal rank-based methods sidestep distributional assumptions of the data.

Created on , Last modified on