Abstract
We introduce INTERLACE, a novel framework that prunes redundant layers in Vision-Language Models (VLMs) while maintaining performance through sample-efficient finetuning. Existing layer pruning methods lead to significant performance drop when applied to VLMs. Instead, we analyze triplets of consecutive layers to identify local redundancy, removing the most redundant of the first two layers, finetuning the remaining layer to compensate for the lost capacity, and freezing the third layer to serve as a stable anchor during finetuning.
By finetuning only a subset of layers on just 1% of the FineVision dataset for one epoch, INTERLACE achieves 88.9% average performance retention after dropping 25% of the network, outperforming alternative pruning methods by 28.4%.
Figure 1. Interlace identifies local redundancy by calculating cosine similarity for triplets of layers. In each selected triplet, the most redundant of the first two layers is dropped (red), the other is fine-tuned (cyan), and the third is frozen as a stable anchor (blue). The performance comparison (top right) shows that Interlace outperforms alternative pruning methods by 28.4%.