Dolphin AI, a Chinese startup specializing in ultrasound-specific medical intelligence, has announced the official release of U2-BENCH—a landmark evaluation standard for multimodal ultrasound AI. This research, recently accepted by ICLR 2026, represents the first systematic attempt to bridge the gap between general AI capabilities and specialized clinical ultrasound requirements.
1. The Challenges of Medical AI: From General Vision to Professional Ultrasound Understanding
Ultrasound imaging is among the most widely used diagnostic tools in global healthcare and continues to play an irreplaceable role in obstetrics and gynecology, emergency medicine, and cardiology. However, automated ultrasound image understanding has long faced significant bottlenecks:
High Variability: Quality is heavily influenced by the operator's technique, leading to substantial fluctuations and numerous artifacts.
Complex Spatial Relationships: Unlike the static slices of CT/MRI, ultrasound presents dynamic structures with strong spatial-contextual relationships.
Lack of Evaluation Systems: While general LVLMs like GPT-4V and Gemini show impressive performance, their professional capabilities in ultrasound have never been systematically evaluated.
To address these challenges, U2-BENCH has been introduced as the first comprehensive benchmark designed to evaluate LVLM capabilities in the ultrasound domain, covering four major task dimensions: classification, detection, regression, and text generation.
2. Core Design: Comprehensive Anatomical Coverage and Clinical-Heuristic Tasks
The core value of U2-BENCH lies in its high clinical relevance and rigorous construction pipeline:
2.1 Unprecedented Data Scale and Diversity
Breadth of Coverage: Aggregates 7,241 cases from 40 authorized datasets, spanning 15 anatomical regions with broad anatomical coverage (including fetus, heart, breast, and thyroid).
Scenario Diversity: Covers 50 clinical use cases to ensure evaluation results accurately reflect a model's performance on the medical frontline.
2.2 Eight Clinical Heuristic Task Categories
U2-BENCH organizes ultrasound understanding into four capability levels and eight specific tasks:
Classification Tasks: Disease Diagnosis (DD), View Recognition and Assessment (VRA).
Detection Tasks: Lesion Localization (LL), Organ Detection (OD), Keypoint Detection (KD).
Regression Tasks: Clinical Value Estimation (CVE).
Generation Tasks: Structured Report Generation (RG), Anatomical Caption Generation (CG).
3. Experimental Validation: Defining the Capabilities and Limitations of SOTA Models
A large-scale evaluation of 23 cutting-edge vision-language models was conducted on U2-BENCH:
3.1 Closed-Source Models Still Lead, but Significant Room for Improvement Remains
Top Performance: Dolphin-V1 ranked first with a total score (U2-Score) of 0.5835, significantly outperforming GPT-5 (0.3250) and Gemini-2.5-Pro (0.2968).
Open-Source Comparison: Among open-source models, DeepSeek-VL2 showed the strongest performance, though a generational gap remains in complex reasoning compared to top-tier closed-source models.
3.2 A Pronounced Gap Between Recognition and Reasoning
Classification vs. Spatial Reasoning: Models perform reasonably well on image-level classification such as Disease Diagnosis (DD), but struggle with spatial-related detection (KD/OD) and regression (CVE) tasks.
Challenges in Report Generation (RG): While the linguistic quality of generated text is high, serious deficiencies remain in medical accuracy and structured compliance.
3.3 Key Conclusion: Scaling Alone is Not the Answer
Diminishing Returns from Parameter Scaling: Comparisons within the Qwen family found that increasing model parameters from 3B to 72B brought steady improvements, but gains were not significant in certain spatial reasoning tasks. This suggests that domain-specific ultrasound training is more effective than simply expanding parameter size.
4. Summary and Outlook: Moving Toward Embodied Medical Intelligence
The successful establishment of U2-BENCH proves that ultrasound AI is undergoing a paradigm shift from "single-task narrow models" toward "all-encompassing foundation models." Looking ahead, U2-BENCH is slated for expansion to include:
Dynamic Video Understanding: Moving from single frames to real-time scanning sequences.
Long-Range Embodied Perception: Integrating with hardware such as robotic arms to achieve automated ultrasound scanning.
U2-BENCH is expected to serve as a vital guide for global medical AI researchers, contributing to the construction of a safer and more professional medical world model.
Media Contact
Company Name: Dolphin AI
Contact Person: Ruier Zhao
Email: Send Email
State: Jiaxing
Country: China
Website: https://dolphin-ai.cn/
