Skip to main content
The NCI Community Hub will be retiring in May 2024. For more information please visit the NCIHub Retirement Page:https://ncihub.cancer.gov/groups/ncihubshutdown/overview
close
eeDAP studies Logo
  • Discoverability Visible
  • Join Policy Restricted
  • Created 13 Jan 2017

eeDAP studies

►

Wiki

HTT update Current

Wiki Main Page

Previous Updates

We are pleased to announce some recent accomplishments of the High-throughput truthing project (HTT).

A list of all previous updates can be found on this page. Our publications and presentations are organized in this Zotero library. Our data is being shared in this GitHub repository.

The HTT project’s objective is to create a validation dataset fit for a regulatory purpose. Pursuing this objective is expected to inform regulatory frameworks and be instructive to others to develop their own validation datasets. For more detailed information about the project and/or to register as a pathologist-data-collector, please visit https://ncihub.org/groups/eedapstudies/.

The HTT project has been accepted as a project of the Truthing and Validation Workgroup of PIcc.

The HTT project was also accepted as a project of the W.H.O. International Collaboration for Cancer Classification and Research (IC3R).

  • The IC3R project is named PADEA (Pathologist Annotation Datasets for Evaluating Algorithms) to allow the effort to grow to other applications beyond TILs evaluation.
  • Link to IC3R “Projects” page.

To contact us with any questions or join our efforts: Contact Us HERE

Updates

July 2022

  • Open-Position: ORISE Fellow
    • Victor Garcia, MD, is transitioning into an FDA/CDRH/DIDSR full-time staff fellow
    • We therefore have an open ORISE Fellow position for the next year (starting 01 October 2022): Statistics and Informatics Support the Assessment of Artificial Intelligence and Machine Learning
  • Actively recruiting now – please distribute! jobDescriptionDigitalPathologyAIML-20220617-2_1.pdf (106 KB, uploaded by Katherine N Elfer 1 year 9 months ago)

* Needed: We are looking for an FDA-Approved Scanner: Aperio AT2 DX

  • Pivotal study slides are in our hands!
    • 58 slides from Stony Brook Medicine. Waiting on WSIs and clinical Metadata
  • 19 slides, WSIs, and clinical metadata from Emory School of Medicine with more in preparation
  • One day, the public health emergency will be declared over. Here is guidance on what happens to devices that fall within pandemic enforcement policies, like WSI scanners: LINK
    • Upcoming Presentations: American Statistical Association’s Joint Statistical Meeting
      • Session: Assessing Agreement and Reader Reliability in Medical Imaging Analysis (organized by Si Wen)
      • August 7th, 2022 in Washington DC
      • Link to Session Info
      • Brandon will present “Exploring ​Pathologist-Pathologist Agreement​ as a Baseline for ​ Algorithm-Pathologist Agreement​”
      • Si will present “Limits of Agreement for an MRMC Study”

    * Published: Hongfei Du, Si Wen, Yufei Guo, Fang Jin, and Brandon D. Gallas, “Single reader between-cases AUC estimator with nested data,” Stat Methods Med Res, p. 9622802221111540, Jul. 2022, doi: 10.1177/09622802221111539.

    June 2022

    • Re-Introduction: Phoebe Qian, ORISE Intern
      • Please see Phoebe’s Admin page
      • Phoebe is rising high school senior returning for her second summer with us!
      • Phoebe is hunting for colleges for Fall 2023 with a focus on computer science
    • R01 Submitted: “Development of optical and digital microscopy hardware, software, and statistical methods to enable computational pathology in low resource environments worldwide”
    • Grant Writers: Kim Blenman (Yale), Brandon Gallas (FDA), Markus Herrmann (MGH)

    May 2022

    • 5/27: Upcoming Presentation to College of American Pathologists (CAP) AI Committee
      • New CAP Committee chaired by Raj Dash
    • Recruiting: FDA/CDRH/OSEL/Division of Imaging Diagnostics and Software Reliability (DIDSR)
      • DIDSR is recruiting at all levels, Full Time Employees and Fellows
    • Published Manuscript: “Development of Training Materials for Pathologists to Provide Machine Learning Validation Data of Tumor-Infiltrating Lymphocytes in Breast Cancer”
      • Authors: Victor Garcia, Katherine Elfer, Dieter Peeters, Anna Ehinger, Bruce Werness, Amy Ly, Xiaxion “Bill” Li, Matthew Hanna, Kim Blenman, Roberto Salgado, Brandon Gallas
      • Published 17 May 2022 in Cancers Special Issue “Tumor Infiltrating Lymphocytes (TIL) in Solid Tumors: Emerging Insights.”
      • Open Access Link
    • Presentation: “Development of Pathologist Training Materials using Consensus Driven Annotations of sTIL Assessment in Breast Cancer”
    • Expert Annotations now on DIDSR/HTT repository and other updates
      • Expert Panel annotations (“camic-expert”) now publicly available on DIDSR/HTT
      • R Markdown and PDF Output used for figures in Garcia, et. al Cancers manuscript also available
    • Published Manuscript: “Three-Way Mixed Effect ANOVA to Estimate MRMC Limits of Agreement”

    April 2022

    • Poster Presentation: “Tools for collecting pathologist annotations and understanding interobserver variability”
      • Presenter: Katherine Elfer
      • American Association for Cancer Research (AACR) Annual Conference 2022
      • Presented 8 April 2022 in New Orleans
      • See Poster Here

    March 2022

    • Submitted Manuscript, “Tools to collect and analyze crowd-source pathologist annotations for validating machine learning algorithms”
      • Elfer K, et. al. submitted to the Journal of Medical Imaging
    • Submitted Manuscript, “Development of Training Materials for Pathologists to Provide Machine Learning Validation Data of Tumor-Infiltrating Lymphocytes in Breast Cancer”
      • Garcia V., et. al. submitted to Cancers Special Issue “Tumor Infiltrating Lymphocytes (TIL) in Solid Tumors: Emerging Insights.”

    February 2022

    • Published Manuscript, “FDA fosters innovative approaches in research, resources and collaboration”

    January 2022

    • Accepted: Letter to the Editor of Nature Machine Intelligence
      • Title: “Research, Resources, and Collaboration: FDA Fosters Innovative Approaches”
      • Accepted and will be published rapidly
    • Pathology Informatics Summit 2022
      • Our abstract was accepted!
      • Title: “Development of Pathologist Training Materials using Consensus Driven Annotations of sTIL Assessment in Breast Cancer”
      • Conference Dates: May 9-12, 2022
      • Where: David L. Lawrence Convention Center and Westin Pittsburgh in Pittsburgh, PA
      • Conference link: https://www.pathologyinformatics.org/pathology-informatics-summit
    • Joint Statistical Meetings 2022
      • Topic Contributed Session: Assessing Agreement and Reader Reliability in Medical Imaging Analysis
      • Speakers and tentative titles:
        • Huiman Barnhart, Duke University. A Review of Assessing Agreement
        • Tongrong Wang, Eli Lily and Company. Sample Size Estimation for Overall Agreement for Multiple Raters with Replicated Measurements
        • Brandon Gallas, FDA/CDRH/OSEL/DIDSR. Exploring Pathologist-Pathologist Agreement as a Baseline for Algorithm-Pathologist Agreement
        • Damien Drubay, Gustave Roussy/INSERM. Estimating the Bias and Precision of Algorithm Predictions from the Pathologist Agreement for Non-Gaussian
        • Data Using Mixed Effect Structured Dispersion Model
        • Si Wen, FDA/CDRH/OSEL/DIDSR. Limits of Agreement for an MRMC Study
      • Session Chair: Evangelos Hytopoulos, iRhythm Technologies.
      • Conference Dates: August 6-11, 2022
      • Where: Washington, DC
      • Conference link: https://ww2.amstat.org/meetings/jsm/2022/
    • New ORISE Fellows
      • Stephanie Jou
        • She is currently working at Emory with Dr. Bill Li.
        • Received a B.S. in Chemistry from Georgia State and has been working at Emory since September on metastatic breast cancer research.
        • Stephanie is sourcing the slides and metadata from Emory for the HTT project’s pivotal study.
    • Kim Blenman, Ph.D., MS (Asst. Prof, Yale University)
      • The focus of this partnership is technology development:
        • Incorporate eeDAP microscope control into the WSI viewer caMicroscope
        • Modularize different task-specific workflows in caMicroscope, including agreement/performance assessment
        • Stand up an annotation server on Precision.FDA (MDDT tool?)
        • As part of the fellowship, Dr. Blenman is recruiting students to participate.
    • Recording Available: Tour of HTTdev Repo
      • Bandon provided a basic walk-through of the public and private HTT Github Repositories including how to access the data and use the data structures.
      • New data of expert annotations on select ROIs has been added to the private repo and will be added to the public repo in conjunction with a future manuscript to the Cancers special issue on TILs
      • Recording and slides available at: https://ncihub.org/groups/eedapstudies/wiki/Presentation2021:TourofHTTData
    • Data Sharing Pilot Study Images
      • This is in addition to caMicroscope and Path.Presenter web-based WSI viewers
      • The pilot study images are hosted by Roberto here: https://www.tilsinbreastcancer.org/pitfalls-2/
      • Roberto also added the following language to the landing page of the website: https://www.tilsinbreastcancer.org
        • All materials on this website, including images, annotations, and similar research and educational data, are anonymized and are therefore freely available to browse, download, and use for scientific and educational purposes. (If you have any issues downloading material, please contact us at below email)Contact us at info@tilsinbreastcancer.org
    • HTT MDDT Proposal Posted
      • Originally submitted to the agency 15 November 2019
      • MDDT page on the eeDAPStudies Wiki – available to the public
      • Brandon wanted to post this with the FDA feedback, but that’s a little harder to get through management approval. As an intermediate step, Brandon will write a short MDDT update summarizing the FDA feedback and how we are addressing it.
      • We’ll be submitting a new proposal in the coming months to address the feedback.
    • TIGER and CATALINA Challenges
      • There are 2 challenges underway on developing AI/ML algorithms to perform the TIL assessment.
      • TIGER Challenge (T.umor I.nfiltratinG. lymphocytes in breast cancE.R.)
        • Organized by the Diagnostic Image Analysis Group (DIAG) of the Radboud University Medical Center (Radboudumc) in Nijmegen (The Netherlands) in collaboration with the TILs Working Group.
          • Timeline
          • January 11, 2022: Release of TIGER training set
          • January 28, 2022: Open leaderboard 1 on the experimental test set
          • February 4, 2022: Open leaderboard 2 on the experimental test set
          • April 13, 2022: End of the challenge
          • April 20, 2022: Announce results on the final test set and awards
        • Link: https://tiger.grand-challenge.org/
      • CATALINA Challenge (Coll.Aborative Ti.l vA.LidatIoN. chA.llenge)

    November 2021

    • Published Manuscript: Dudgeon, Sarah N. et. al. “A pathologist-annotated dataset for validating artificial intelligence: a project description and pilot study.” J. Pathol. Inform. 1.45 (2021).
    • Manuscripts in-progress
      • Nature Machine Intelligence Correspondence: Research, Resources, and Collaboration: FDA Fosters Innovative Approaches
      • Cancers Special Issue on Tumor Infiltrating Lymphocytes (TIL) in Solid Tumors: Training Development for the Evaluation of TILs in Breast Cancer
      • Cancers Special Issue on Tumor Infiltrating Lymphocytes (TIL) in Solid Tumors: HTT Data Analysis Methods
      • SPIE Proceedings Paper: HTT Pilot Study informed Improvements on Technology for Data-Collection
    • Data Collection:
      • Completed 8 hours of discussions about sTILs annotations with HTT panel of experts.
      • The eeDAP microscope system is set up at Yale. We are looking for pathologists to support the project by assessing the tumor-infiltrating lymphocytes in breast cancer slides. Click here for more information.
    • Abstract submitted to 2022 American Association of Cancer Research.
      • Title: Pilot study to develop tools for high throughput truthing of pathologist annotations to validate AI/ML algorithms in digital pathology
      • Conference Dates: April 8-13, 2022
    • Abstract Accepted: Abstract for the 2022 SPIE Medical Imaging Conference has been accepted for oral presentation.
      • Title: Quantitative assessment of pathologist agreement using High Throughput Truthing (HTT)
      • Presentation Date: 21 February 2022

    October 2021

    • 29 October 2021: PIcc provided update to CDRH Leadership Team
    • 21 October 2021: PIcc featured at FDA/CDRH All Hands
      • Joe Lennerz (PIcc Steering Committee Director), Brandon Gallas (PIcc-CDRH Liaison and Co-Chair of the Truthing & Validation Working Group), and Jithesh Viteel (PIcc-Conveener Representative) were panelists in a Collaborative Community Town Hall
    • Abstract submitted to 2022 Pathology Informatics Summit
      • Title: Development of Pathologist Training Materials using Consensus Driven Annotations of sTIL Assessment in Breast Cancer
      • Conference Dates: May 9-12, 2022
    • 01 October 2021: Start of award year for the FDA Office of Women’s Health grant funding
      • Hire: Victor Garcia, M.D. is a current Oak Ride Institute for Science and Education (ORISE) Fellow at the U.S. Food and Drug Administration, board-certified in Internal Medicine, and a board-eligible Clinical Informatician. Dr. Garcia contributes to data collection efforts by coordinating annotation sessions with our expert pathologists, sourcing glass slides, developing clinical training manuals for annotating sTILs in breast cancer, and driving our HTT data set MDDT submission efforts. Dr. Garcia also holds a volunteer position with Stony Brook Medicine.
      • Hire: Part-Time employee at Emory University to aid in our slide-sourcing efforts.
      • There is moderate budget for data-collection
    • October 2021: Revisions made to the primary eeDAP Studies Wiki Pages

    Created on , Last modified on