We are pleased to announce some recent accomplishments of the High-throughput truthing project (HTT).
A list of all previous updates can be found on this page. Our publications and presentations are organized in this Zotero library. Our data is being shared in this GitHub repository.
The HTT project’s objective is to create a validation dataset fit for a regulatory purpose. Pursuing this objective is expected to inform regulatory frameworks and be instructive to others to develop their own validation datasets. For more detailed information about the project and/or to register as a pathologist-data-collector, please visit https://ncihub.org/groups/eedapstudies/.
The HTT project has been accepted as a project of the Truthing and Validation Workgroup of PIcc.
- PIcc is The Pathology Innovation Collaborative Community.
- PIcc is pronounced “Pie” CC.
- Link to the Truthing and Validation workgroup.
- Link to the FDA Collaborative Communities Program.
The HTT project was also accepted as a project of the W.H.O. International Collaboration for Cancer Classification and Research (IC3R).
- The IC3R project is named PADEA (Pathologist Annotation Datasets for Evaluating Algorithms) to allow the effort to grow to other applications beyond TILs evaluation.
- Link to IC3R “Projects” page.
To contact us with any questions or join our efforts: Contact Us HERE
Updates
May 2022
- 5/27: Upcoming Presentation to College of American Pathologists (CAP) AI Committee
- New CAP Committee chaired by Raj Dash
- Recruiting: FDA/CDRH/OSEL/Division of Imaging Diagnostics and Software Reliability (DIDSR)
- DIDSR is recruiting at all levels, Full Time Employees and Fellows
- Published Manuscript: “Development of Training Materials for Pathologists to Provide Machine Learning Validation Data of Tumor-Infiltrating Lymphocytes in Breast Cancer”
- Authors: Victor Garcia, Katherine Elfer, Dieter Peeters, Anna Ehinger, Bruce Werness, Amy Ly, Xiaxion “Bill” Li, Matthew Hanna, Kim Blenman, Roberto Salgado, Brandon Gallas
- Published 17 May 2022 in Cancers Special Issue “Tumor Infiltrating Lymphocytes (TIL) in Solid Tumors: Emerging Insights.”
- Open Access Link
- Presentation: “Development of Pathologist Training Materials using Consensus Driven Annotations of sTIL Assessment in Breast Cancer”
- Presenter: Victor Garcia
- Pathology Informatics Summit 2022
- Presented 11 May 2022 in Pittsburgh, PA
- View Abstract & Slides Here
- Expert Annotations now on DIDSR/HTT repository and other updates
- Expert Panel annotations (“camic-expert”) now publicly available on DIDSR/HTT
- R Markdown and PDF Output used for figures in Garcia, et. al Cancers manuscript also available
- Published Manuscript: “Three-Way Mixed Effect ANOVA to Estimate MRMC Limits of Agreement”
- Authors: Si Wen and Brandon Gallas.
- Published 11 May 2022 in Statistics of Biopharmaceutical Research
- https://doi.org/10.1080/19466315.2022.2063169
April 2022
- Tutorial by Brandon Gallas : “Tutorial on Reader Study Designs and MRMC Analysis”
- Brandon presented at an FDA Internal Training on 8 April 2022
- Slides Available: https://ncihub.org/groups/eedapstudies/wiki/DeviceAdviceAIMLImaging
- Presentation info on eeDAP Studies Zotero
- Poster Presentation: “Tools for collecting pathologist annotations and understanding interobserver variability”
- Presenter: Katherine Elfer
- American Association for Cancer Research (AACR) Annual Conference 2022
- Presented 8 April 2022 in New Orleans
- See Poster Here
March 2022
- FDA-PIcc Discussion Series: Can AI Grand-Challenges inform Regulatory Science in Anatomic Pathology?
- Speakers: Roberto Salgado, MD and Francesco Ciompi, PhD
- Project Page: https://pathologyinnovationcc.org/projects/project-ai-grand-challenges
- Submitted Manuscript, “Tools to collect and analyze crowd-source pathologist annotations for validating machine learning algorithms”
- Elfer K, et. al. submitted to the Journal of Medical Imaging
- Submitted Manuscript, “Development of Training Materials for Pathologists to Provide Machine Learning Validation Data of Tumor-Infiltrating Lymphocytes in Breast Cancer”
- Garcia V., et. al. submitted to Cancers Special Issue “Tumor Infiltrating Lymphocytes (TIL) in Solid Tumors: Emerging Insights.”
February 2022
- Published Manuscript, “FDA fosters innovative approaches in research, resources and collaboration”
- Authors: Brandon D. Gallas, Aldo Badano, Sarah Dudgeon, Katherine Elfer, Victor Garcia, Jochen K. Lennerz, Kyle Myers, Nicholas Petrick & Ed Margerrison
- Link to Journal: https://www.nature.com/articles/s42256-022-00450-2
- Link to shareable online viewable copy: https://rdcu.be/cHClY
- Published 23 February 2022