Skip to main content
The NCI Community Hub will be retiring in May 2024. For more information please visit the NCIHub Retirement Page:https://ncihub.cancer.gov/groups/ncihubshutdown/overview
close

HTT update 20220126

Wiki Main Page

Previous Updates

We are pleased to announce some recent accomplishments of the High-throughput truthing project (HTT).

A list of all previous updates can be found on this page. Our publications and presentations are organized in this Zotero library. Our data is being shared in this GitHub repository.

The HTT project’s objective is to create a validation dataset fit for a regulatory purpose. Pursuing this objective is expected to inform regulatory frameworks and be instructive to others to develop their own validation datasets. For more detailed information about the project and/or to register as a pathologist-data-collector, please visit https://ncihub.org/groups/eedapstudies/.

The HTT project has been accepted as a project of the Truthing and Validation Workgroup of PIcc.

The HTT project was also accepted as a project of the W.H.O. International Collaboration for Cancer Classification and Research (IC3R).

  • The IC3R project is named PADEA (Pathologist Annotation Datasets for Evaluating Algorithms) to allow the effort to grow to other applications beyond TILs evaluation.
  • Link to IC3R “Projects” page.

To contact us with any questions or join our efforts: Contact Us HERE


Updates

January 2022

  • Accepted: Letter to the Editor of Nature Machine Intelligence
    • Title: “Research, Resources, and Collaboration: FDA Fosters Innovative Approaches”
    • Accepted and will be published rapidly
  • Pathology Informatics Summit 2022
    • Our abstract was accepted!
    • Title: “Development of Pathologist Training Materials using Consensus Driven Annotations of sTIL Assessment in Breast Cancer”
    • Conference Dates: May 9-12, 2022
    • Where: David L. Lawrence Convention Center and Westin Pittsburgh in Pittsburgh, PA
    • Conference link: https://www.pathologyinformatics.org/pathology-informatics-summit
  • Joint Statistical Meetings 2022
    • Topic Contributed Session: Assessing Agreement and Reader Reliability in Medical Imaging Analysis
    • Speakers and tentative titles:
      • Huiman Barnhart, Duke University. A Review of Assessing Agreement
      • Tongrong Wang, Eli Lily and Company. Sample Size Estimation for Overall Agreement for Multiple Raters with Replicated Measurements
      • Brandon Gallas, FDA/CDRH/OSEL/DIDSR. Exploring Pathologist-Pathologist Agreement as a Baseline for Algorithm-Pathologist Agreement
      • Damien Drubay, Gustave Roussy/INSERM. Estimating the Bias and Precision of Algorithm Predictions from the Pathologist Agreement for Non-Gaussian
      • Data Using Mixed Effect Structured Dispersion Model
      • Si Wen, FDA/CDRH/OSEL/DIDSR. Limits of Agreement for an MRMC Study
    • Session Chair: Evangelos Hytopoulos, iRhythm Technologies.
    • Conference Dates: August 6-11, 2022
    • Where: Washington, DC
    • Conference link: https://ww2.amstat.org/meetings/jsm/2022/
  • New ORISE Fellows
    • Stephanie Jou
      • She is currently working at Emory with Dr. Bill Li.
      • Received a B.S. in Chemistry from Georgia State and has been working at Emory since September on metastatic breast cancer research.
      • Stephanie is sourcing the slides and metadata from Emory for the HTT project’s pivotal study.
  • Kim Blenman, Ph.D., MS (Asst. Prof, Yale University)
    • The focus of this partnership is technology development:
      • Incorporate eeDAP microscope control into the WSI viewer caMicroscope
      • Modularize different task-specific workflows in caMicroscope, including agreement/performance assessment
      • Stand up an annotation server on Precision.FDA (MDDT tool?)
      • As part of the fellowship, Dr. Blenman is recruiting students to participate.
  • Recording Available: Tour of HTTdev Repo
    • Bandon provided a basic walk-through of the public and private HTT Github Repositories including how to access the data and use the data structures.
    • New data of expert annotations on select ROIs has been added to the private repo and will be added to the public repo in conjunction with a future manuscript to the Cancers special issue on TILs
    • Recording and slides available at: https://ncihub.org/groups/eedapstudies/wiki/Presentation2021:TourofHTTData
  • Data Sharing Pilot Study Images
    • This is in addition to caMicroscope and Path.Presenter web-based WSI viewers
    • The pilot study images are hosted by Roberto here: https://www.tilsinbreastcancer.org/pitfalls-2/
    • Roberto also added the following language to the landing page of the website: https://www.tilsinbreastcancer.org
      • All materials on this website, including images, annotations, and similar research and educational data, are anonymized and are therefore freely available to browse, download, and use for scientific and educational purposes. (If you have any issues downloading material, please contact us at below email)Contact us at info@tilsinbreastcancer.org
  • HTT MDDT Proposal Posted
    • Originally submitted to the agency 15 November 2019
    • MDDT page on the eeDAPStudies Wiki – available to the public
    • Brandon wanted to post this with the FDA feedback, but that’s a little harder to get through management approval. As an intermediate step, Brandon will write a short MDDT update summarizing the FDA feedback and how we are addressing it.
    • We’ll be submitting a new proposal in the coming months to address the feedback.
  • TIGER and CATALINA Challenges
    • There are 2 challenges underway on developing AI/ML algorithms to perform the TIL assessment.
    • TIGER Challenge (T.umor I.nfiltratinG. lymphocytes in breast cancE.R.)
      • Organized by the Diagnostic Image Analysis Group (DIAG) of the Radboud University Medical Center (Radboudumc) in Nijmegen (The Netherlands) in collaboration with the TILs Working Group.
        • Timeline
        • January 11, 2022: Release of TIGER training set
        • January 28, 2022: Open leaderboard 1 on the experimental test set
        • February 4, 2022: Open leaderboard 2 on the experimental test set
        • April 13, 2022: End of the challenge
        • April 20, 2022: Announce results on the final test set and awards
      • Link: https://tiger.grand-challenge.org/
    • CATALINA Challenge (Coll.Aborative Ti.l vA.LidatIoN. chA.llenge)

November 2021

  • Published Manuscript: Dudgeon, Sarah N. et. al. “A pathologist-annotated dataset for validating artificial intelligence: a project description and pilot study.” J. Pathol. Inform. 1.45 (2021).
  • Manuscripts in-progress
    • Nature Machine Intelligence Correspondence: Research, Resources, and Collaboration: FDA Fosters Innovative Approaches
    • Cancers Special Issue on Tumor Infiltrating Lymphocytes (TIL) in Solid Tumors: Training Development for the Evaluation of TILs in Breast Cancer
    • Cancers Special Issue on Tumor Infiltrating Lymphocytes (TIL) in Solid Tumors: HTT Data Analysis Methods
    • SPIE Proceedings Paper: HTT Pilot Study informed Improvements on Technology for Data-Collection
  • Data Collection:
    • Completed 8 hours of discussions about sTILs annotations with HTT panel of experts.
    • The eeDAP microscope system is set up at Yale. We are looking for pathologists to support the project by assessing the tumor-infiltrating lymphocytes in breast cancer slides. Click here for more information.
  • Abstract submitted to 2022 American Association of Cancer Research.
    • Title: Pilot study to develop tools for high throughput truthing of pathologist annotations to validate AI/ML algorithms in digital pathology
    • Conference Dates: April 8-13, 2022
  • Abstract Accepted: Abstract for the 2022 SPIE Medical Imaging Conference has been accepted for oral presentation.
    • Title: Quantitative assessment of pathologist agreement using High Throughput Truthing (HTT)
    • Presentation Date: 21 February 2022

October 2021

  • 29 October 2021: PIcc provided update to CDRH Leadership Team
  • 21 October 2021: PIcc featured at FDA/CDRH All Hands
    • Joe Lennerz (PIcc Steering Committee Director), Brandon Gallas (PIcc-CDRH Liaison and Co-Chair of the Truthing & Validation Working Group), and Jithesh Viteel (PIcc-Conveener Representative) were panelists in a Collaborative Community Town Hall
  • Abstract submitted to 2022 Pathology Informatics Summit
    • Title: Development of Pathologist Training Materials using Consensus Driven Annotations of sTIL Assessment in Breast Cancer
    • Conference Dates: May 9-12, 2022
  • 01 October 2021: Start of award year for the FDA Office of Women’s Health grant funding
    • Hire: Victor Garcia, M.D. is a current Oak Ride Institute for Science and Education (ORISE) Fellow at the U.S. Food and Drug Administration, board-certified in Internal Medicine, and a board-eligible Clinical Informatician. Dr. Garcia contributes to data collection efforts by coordinating annotation sessions with our expert pathologists, sourcing glass slides, developing clinical training manuals for annotating sTILs in breast cancer, and driving our HTT data set MDDT submission efforts. Dr. Garcia also holds a volunteer position with Stony Brook Medicine.
    • Hire: Part-Time employee at Emory University to aid in our slide-sourcing efforts.
    • There is moderate budget for data-collection
  • October 2021: Revisions made to the primary eeDAP Studies Wiki Pages

Created on , Last modified on