MedCLIP-SAMv2: Towards Universal Text-Driven Medical Image Segmentation

Segmentation of anatomical structures and pathologies in medical images is essential for modern disease diagnosis, clinical research, and treatment planning. While significant advancements have been made in deep learning-based segmentation techniques, many of these methods still suffer from limitations in data efficiency, generalizability, and interactivity. Recently, foundation models like CLIP and Segment-Anything-Model (SAM) have paved the way for interactive and universal image segmentation. In this work, we introduce MedCLIP-SAMv2, a framework that integrates BiomedCLIP and SAM to perform text-driven medical image segmentation in zero-shot and weakly supervised settings. The approach fine-tunes BiomedCLIP with a new DHN-NCE loss and leverages M2IB to create visual prompts for SAM; we also explore uncertainty-aware refinement via checkpoint ensembling.

We evaluate on four modalities/tasks: breast tumor ultrasound, brain tumor MRI, lung X-ray, and lung CT. Below are the main comparison and ablation tables.

We present qualitative segmentation results produced by MedCLIP-SAMv2 across diverse imaging modalities, including brain MRI, breast ultrasound, chest X-ray, and chest CT. The results demonstrate that text-driven prompts, combined with BiomedCLIP-based saliency and SAM-based refinement, yield accurate and coherent segmentations that align well with anatomical structures and pathological regions, even in the absence of pixel-level supervision.

We compare text-driven saliency maps generated using BiomedCLIP and standard CLIP across four representative medical imaging modalities: brain MRI, breast ultrasound, chest X-ray, and chest CT. BiomedCLIP consistently produces more localized and anatomically meaningful responses aligned with clinically relevant regions, while standard CLIP often exhibits diffuse or semantically ambiguous activations.