Localized Conformal Prediction for Image Classification with Vision-Language Models

Conformal predictions have attracted significant attention in the field of uncertainty quantification, mainly because of their strong marginal coverage guarantees. Full conditional guarantee is not an attainable goal, a well known fact in conformal predictions literature. As a result, several approaches have tried to approximate this behavior by adapting the conformal sets of test-time samples according to their similarity to calibration examples. Although the latter has gained traction and shown impressive performances for regression problems, its application to image classification remains under-explored. We conduct an extensive benchmarking on natural image classification tasks with vision-language models (VLMs), using our open source implementation of a recent localized conformal prediction algorithm. We show that straightforward usage of the cosine similarity between test-time and calibration visual features, an intuitive choice for VLMs, is not sufficient to improve over the non-local baselines. In response, we propose a simple non-linear transformation of the cosine similarities, which conserves marginal coverage guarantees and achieves statistically significant mean set sizes reduction. Code is available at github.com/cfuchs2023/lcpvlm/.

Affiliations

UCLouvainSST/ICTM/ELEN - Pôle en ingénierie électrique

Citations

APA
Chicago
FWB

Fuchs, C., Bary, T., & Macq, B. (2025). Localized Conformal Prediction for Image Classification with Vision-Language Models. 13th European Workshop on Visual Information Processing (EUVIP 2025), La Valette, Malte. https://hdl.handle.net/2078.5/248959