Evi-Steer: Learning to Steer Biomedical Vision-Language Models through Efficient and Generalizable Evidential Tuning

Parameter-efficient adaptation of biomedical vision-language foundation models is essential for robust multimodal understanding in low-data and shifted clinical settings. Existing adaptation methods are often deterministic and can apply residual updates even when image-text evidence is ambiguous or unreliable. We present Evi-Steer, an evidential cross-modal low-dimensional steering framework for BiomedCLIP that enables uncertainty-aware fine-tuning while updating only 0.11% of the total model parameters. Evi-Steer performs lightweight token updates in both the vision and text encoders, estimates latent-dimension epistemic uncertainty, and uses these estimates to conservatively gate residual steering. It further introduces Dempster-Shafer cross-modal confidence fusion, conditioning visual adaptation on textual confidence to suppress conflicting or uncertain updates. Across 15 biomedical imaging datasets spanning 8 organs and 8 imaging modalities, Evi-Steer improves few-shot learning and domain generalization, providing a practical pathway for reliable biomedical VLM adaptation.

• Evidential Low-Dimensional Representation Steering: Evi-Steer projects text and vision activations into a compact latent space, computes lightweight token updates, and estimates latent-dimension epistemic uncertainty in a single forward pass.
• Confidence-Weighted Residual Updates: Evidence-derived uncertainty is converted into confidence weights, allowing the model to steer strongly when evidence is reliable and adapt conservatively when evidence is weak.
• Cross-Modal Reliability Fusion: A Dempster-Shafer belief fusion mechanism conditions vision-side updates on textual confidence, suppressing ambiguous or conflicting image-text updates under domain shift.
• Frozen BiomedCLIP Backbone: The core BiomedCLIP vision and text encoders remain frozen, preserving pretrained representations while training a small set of steering modules.

We conduct ablation studies to better understand the contribution of the proposed components. Results show that visual adaptation, textual adaptation, evidential uncertainty modeling, and cross-modal belief fusion all contribute to the final domain-generalization performance, with visual adaptation having the largest impact on out-of-distribution robustness.

We further analyze the effect of the intervention depth d and the latent adapter dimension r. Increasing the number of adapted layers consistently improves domain generalization performance, indicating that distributing lightweight steering modules throughout the network is more effective than shallow adaptation. We also observe that performance improves as the adapter dimension increases from small values and reaches its optimum at r = 4. Larger dimensions provide no additional benefit and slightly reduce performance, suggesting that compact low-dimensional updates achieve the best balance between adaptation capacity and overfitting.