Boosting vision-language models for histopathology classification: Predict all at once
(2024) Foundation Models for General Medical AI — Location: Marrakesh, Morocco (6.October.2024)
Files
240901883v1.pdf
Open Access - Adobe PDF
- 6.09 MB
Details
- Authors
- Abstract
- The development of vision-language models (VLMs) for histo-pathology has shown promising new usages and zero-shot performances. However, current approaches, which decompose large slides into smaller patches, focus solely on inductive classification, i.e., prediction for each patch is made independently of the other patches in the target test data. We extend the capability of these large models by introducing a transductive approach. By using text-based predictions and affinity relationships among patches, our approach leverages the strong zero-shot capabilities of these new VLMs without any additional labels. Our experiments cover four histopathology datasets and five different VLMs. Operating solely in the embedding space (i.e., in a black-box setting), our approach is highly efficient, processing 100,000 patches in just a few seconds, and shows significant accuracy improvements over inductive zero-shot classification. Code available at https://github.com/FereshteShakeri/Histo-TransCLIP.
- Affiliations
Citations
Zanella, M., Shakeri, F., Huang, Y., Bahig, H., & Ben Ayed, I. (2024). Boosting vision-language models for histopathology classification: Predict all at once. Lecture Notes in Computer Science, 15184(1), 153-162. https://hdl.handle.net/2078.5/267815 (Original work published 2024)