This paper establishes a systematic protocol for 'stitching' heterogeneous Vision Foundation Models (e.g., CLIP and DINOv2) to share early layers while retaining specialized capabilities.
arXiv · March 16, 2026 · 2603.12433
Why it matters
Instead of training massive multi-modal models from scratch, this enables the modular composition of existing pre-trained models. The VFM Stitch Tree (VST) provides a new way to manage accuracy-latency trade-offs in multi-modal LLM deployment.
From the abstract
Model stitching, connecting early layers of one model (source) to later layers of another (target) via a light stitch layer, has served as a probe of representational compatibility. Prior work finds that models trained on the same dataset remain stitchable (negligible accuracy drop) despite different initializations or objectives. We revisit stitching for Vision Foundation Models (VFMs) that vary in objectives, data, and modality mix (e.g., CLIP, DINOv2, SigLIP 2) and ask: Are heterogeneous VFMs