BiomedCoOp

Abstract

Recent advancements in vision-language models (VLMs), such as CLIP, have demonstrated substantial success in self-supervised representation learning for vision tasks. However, effectively adapting VLMs to downstream applications remains challenging, as their accuracy often depends on time-intensive and expertise-demanding prompt engineering, while full model fine-tuning is costly. This is particularly true for biomedical images, which, unlike natural images, typically suffer from limited annotated datasets, unintuitive image contrasts, and nuanced visual features. Recent prompt learning techniques, such as Context Optimization (CoOp) intend to tackle these issues, but still fall short in generalizability. Meanwhile, explorations in prompt learning for biomedical image analysis are still highly limited. In this work, we propose BiomedCoOp, a novel prompt learning framework that enables efficient adaptation of BiomedCLIP for accurate and highly generalizable few-shot biomedical image classification. Our approach achieves effective prompt context learning by leveraging semantic consistency with average prompt ensembles from Large Language Models (LLMs) and knowledge distillation with a statistics-based prompt selection strategy. We conducted comprehensive validation of our proposed framework on 11 medical datasets across 9 modalities and 10 organs against existing state-of-the-art methods, demonstrating significant improvements in both accuracy and generalizability.

Method

LLM-guided Prompt Ensembles: LLM-generated biomedical descriptions guide context learning.
Semantic Consistency: Prompt contexts are aligned with averaged LLM embeddings.
Selective Knowledge Distillation: Statistics-based pruning removes noisy prompts.
BiomedCLIP Backbone: Enables robust multi-modal biomedical representations.

Few-shot Evaluation

Method	K = 1	K = 2	K = 4	K = 8	K = 16
Zero-shot Methods
BiomedCLIP	–	–	42.05	–	–
BiomedCLIP + Ensemble	–	–	52.27	–	–
BiomedCLIP + Selective Ensemble	–	–	53.72	–	–
CLIP-based Adapter Methods
CLIP-Adapter	44.66 ± 2.97	43.91 ± 2.48	44.36 ± 1.94	45.42 ± 2.38	46.69 ± 1.71
Tip-Adapter	49.19 ± 4.84	52.36 ± 6.57	57.33 ± 5.07	61.98 ± 5.76	67.15 ± 4.25
Tip-Adapter-F	51.17 ± 8.33	52.74 ± 5.88	61.23 ± 6.22	65.91 ± 3.64	70.91 ± 2.65
Prompt Learning Methods
CoOp	50.16 ± 6.93	54.18 ± 4.31	59.75 ± 3.72	65.84 ± 3.66	69.62 ± 2.83
CoCoOp	48.49 ± 4.39	51.28 ± 5.06	54.69 ± 4.79	61.08 ± 3.49	65.09 ± 2.87
KgCoOp	50.85 ± 5.59	53.18 ± 4.33	57.82 ± 4.50	62.08 ± 2.59	62.84 ± 1.72
ProGrad	51.88 ± 6.39	54.71 ± 4.46	60.42 ± 4.78	65.61 ± 3.02	67.13 ± 3.00
BiomedCoOp (Ours)	57.03 ± 2.80	59.13 ± 3.64	63.95 ± 2.42	68.32 ± 2.65	72.42 ± 1.69

Base-to-Novel Generalization

Dataset	Split	BiomedCLIP	CoOp	CoCoOp	KgCoOp	ProGrad	BiomedCoOp
Average	Base	47.84	73.85	72.26	68.36	71.67	76.26
	Novel	65.42	64.75	67.03	64.08	66.93	73.92
	HM	53.81	67.23	67.22	64.61	67.43	75.07

Visual Interpretability (Saliency Maps)

We visualize model attention using gScoreCAM. Compared to baseline prompt learning methods, BiomedCoOp consistently focuses on clinically relevant regions while suppressing background noise across modalities.

Saliency map comparison across methods — Comparison of gScoreCAM saliency maps for BiomedCLIP, CoOp, CoCoOp, and **BiomedCoOp (ours)** across representative biomedical images.

BibTeX

@inproceedings{koleilat2025biomedcoop,
  title={Biomedcoop: Learning to prompt for biomedical vision-language models},
  author={Koleilat, Taha and Asgariandehkordi, Hojat and Rivaz, Hassan and Xiao, Yiming},
  booktitle={Proceedings of the Computer Vision and Pattern Recognition Conference},
  pages={14766--14776},
  year={2025}
}