Inspirata Team
Tampa, FL
- Lead: James Monaco, PhD
- Richard Morroney, BS
- Santosh Bhargava, PhD
- Mark Lloyd, PhD
Here are our objectives and plans:
- Our interests are largely in the image analysis domain, including the detection/classification of mitotic figures.
- We are interested in collecting data on mitotic figures in breast specimens for training our classifier. We would like to include reader variability in this, recognizing that pathologists are not in total agreement on which cells are mitotic figures. Some cells are have high agreement and others do not.
- We can easily get 7 or more pathologists that could participate in a reader study. We would like the majority to be breast pathologists.
- We can get the breast biopsy specimens and will likely have long-term recurrence information. It would be great to think of a way to factor this in.
- We have done and are interested in doing ROC-like studies to collect data, but we are unsure how to use this information to better evaluate reader agreement and validate our classifier.
- We have access to hardware that could be configured into an eeDAP system and could partner with a site for data collection.
- We like the idea of using eeDAP in someway since I think the mitotic scores from reading with the microscope will be more accurate than those from the images. It would be interesting to investigate this, and it would be valuable to us from an algorithm standpoint since it may indicate the limitations on what will be possible.
Follow up Discussion
Brandon
Comparing pathologists to each other is my main focus. I am analyzing, developing, and tailoring methods for a data set right now. Many readers counting mitoses following clinical protocol. It is messy. Concordance is moderate, variability is high. I don’t have all the answers, but I’m trying to engage my statistician colleagues to help with this problem and I’m hoping this interaction (eeDAP studies group) will get more discussion and consensus. Do you have a data set already? My NIH colleague is not quite warmed up to sharing the results and discussing in the group yet, but I’m working on him. If someone else put something similar on the table that might help. If you made a direct plea for help on this, that could draw him out too.
Long-term recurrence is an outcome that allows for an evaluation of prognostic ability. Tools for that evaluation are more familiar to me. We have some survival data for the data mentioned above, and we have done those analyses (Harrell’s C-stat, ROC, Sens, Spec).
James:
I am including an initial draft of the experiment MitosisCollectionFDA.docx (77 KB, uploaded by Brandon D. Gallas 7 years 1 month ago). As I’ve never really written formal protocols for such experiments, feel free to point out all the deficiencies.
Brandon:
I have read your draft and it is a good start summarizing the data collection. If I was forced to discuss “deficiencies”, I would point out the lack of objectives, endpoints, and analyses. That said, I would love to have this discussion with the others in the group. Otherwise, it might look like I am helping one company instead of having a scientific discussion in the working group.
Can I share the draft experiment with the group when we share summaries of objectives and plans?
James:
Feel free to share with anyone. There is nothing to hide here, and lots of benefit from getting others opinions.
You make a good point as to objectives, etc… I left these out since I wanted to see exactly what data I would be using before making them. However, it’s easy to share some questions I’m interested in if we get the recurrence data I expect:
1) How well does mitotic score predict recurrence? 2) How well does mitotic count predict recurrence? 3) How well do consensus measures (both of score and count) predict recurrence? 4) What is the inter-observer variability of both the scoring and the counting? 5) What is the relationship between the score and the count?
Also, I have questions I need to answer about my algorithm:
1) How does it compare to pathologists in terms of ability to detect mitoses and ability to predict recurrence using its mitotic detections?
Finally, this experiment is important for both training a mitotic detection algorithm and evaluating it. My underlying hypothesis is that consensus scores are what we should consider the “ground truth”. We can confirm this if the consensus better predicts recurrence.
Actually, your comments made me rethink things. I think I should revise the experiment (see attached) so that each box (previously a 2000×2000 cutout) is replaced by the entire rectangle (which are about 4000×4000). This would increase the time needed for analysis, but it would make future analysis more insightful. Anyway, this could be something to discuss.