Skip to main content

Presentation 2022: Pathology Innovation Collaborative Community Webinar - ROC and MRMC Tutorials - Gallas

Back to Presentations and Publications Main Page

Pathology Innovation Collaborative Community Webinar

  • Webinar
  • 12:00 PM, Friday, 5 August 2022

This presentation was also give as FDA internal training on 8 April 2022.

Title 1: Tutorial on Reader Study Designs and MRMC Analysis

Title 2: ROC curves: Receiver Operating Characteristic Curves




In this talk I will present the major elements to design, execute, and analyze reader studies. In a reader study, clinicians perform an objective task given medical images. The purpose of a reader study is to assess the clinicians’ performance doing the task given the images or to compare performance given images from a new technology to the performance given images from the reference technology. The clinician is part of the technology assessment. The clinicians are the readers and the medical images are the cases. The objective tasks are to make diagnostic evaluations or other measurements given the images (aligned with the intended use of the images). Some refer to the evaluations as subjective because they involve humans who are biased and not fully reproducible. I prefer to refer to the evaluations as objective to distinguish them from evaluations that are, in fact, opinions that have no right answer. An objective evaluation can be compared to truth if truth is available. Of course, there is bias and variability from human evaluations. As such, analysis methods for reader performance (variance estimates, confidence intervals, hypothesis tests) need to account for such bias and variability from the readers while they also account for the bias and variability from the cases. Such an analysis is referred to as a multi-reader, multi-case MRMC analysis. An MRMC analysis can summarize average reader performance that is expected to generalize to the population of readers and the population of cases.


Brandon D. Gallas, PhD

  • Division of Imaging, Diagnostics, and Software Reliability, OSEL/CDRH/FDA, White Oak, MD

Created on , Last modified on