Text Capability Loss in Vision-Language Adaptation: An Attention-Sink Diagnosis

A base LLM concentrates attention on an early sink position; after VL fine-tuning the sink is preserved with per-head QK-RMSNorm but corrupted without it, and instruction following degrades with it. — A pretrained LLM concentrates attention on an early sink position. Whether that sink survives VL fine-tuning varies sharply across language backbones, and instruction-following ability follows it down.

TL;DR

Turning an LLM into a vision-language model can corrupt its attention sink and erode instruction following. Sink Strength, a single number read off the base LLM before any vision training, predicts how much a backbone will lose (Spearman ρ = 0.97 on the six-pair diagnostic set and 0.88 on a 17-pair extension). Theorem 1 shows how per-head QK-RMSNorm removes explicit raw-input-magnitude sensitivity from the sink-gap bound, while the layerwise variant lacks the same per-head guarantee.

Abstract

Fine-tuning a pretrained LLM into a vision-language model (VLM) can erode the backbone's text capability, with the damage concentrated on tasks that require following exact output rules, such as instruction following and chain-of-thought reasoning graded on a strictly parsed final answer.

We trace this gap to attention-sink corruption. VL fine-tuning perturbs the early sink position that anchors a large fraction of attention probability, and how well the base LLM preserves its sink determines how much of the affected capability survives adaptation.

Building on this view, we introduce Sink Strength, a single scalar computed on the base LLM that predicts post-VL degradation without any VL training. It consistently tracks relative degradation across the six VLM–LLM pairs and multiple format-sensitive tasks.

Complementing this diagnostic, post-pretraining QK-RMSNorm injection fails to reproduce the protection of native QK-RMSNorm, while several off-the-shelf weight-merging settings fail to recover the lost capability after VL training.

Method

Attention-sink corruption. A pretrained LLM concentrates a large share of attention on a few early sink positions. VL fine-tuning perturbs this sink, and format-sensitive text ability degrades with it.
Sink Strength (S). A training-free scalar, the median sink-logit lead over the last ~10 decoder layers, measured on the base LLM from 15 calibration prompts in inference-only forward passes. No VL training and no labels required.
Strong-sink criterion. Appendix D.2 uses the strict rule G_base > 2, equivalently a_sink > 0.881, over non-degenerate late-layer prompt × layer × head × query observations. This gives 8.5% prevalence for Molmo2-O and 58% for Qwen3-8B.
Why per-head QK-RMSNorm protects (Theorem 1). Under the paper's perturbation assumptions, per-head query and key normalization removes explicit raw-input-magnitude sensitivity from the sink-gap bound. The layerwise variant pools energy across heads and does not provide the same per-head guarantee.

Results

Sink Strength, measured on the reference LLM without VL training, ranks the five headline backbones plus the Molmo2-O control by their VLM-LM/reference-LLM IFEval gap (Spearman ρ = 0.97). Across the four format-sensitive tasks, the six-pair correlations remain between 0.88 and 0.97. The IFEval result extends to ρ = 0.88 on 17 VLM–LLM pairs spanning dense and MoE architectures, 1.5B–32B backbones, and Qwen, Olmo, OLMoE, Mistral, and Llama families.

Scatter of Sink Strength versus IFEval delta between each VLM language model and its reference LLM for six pairs, Spearman rho 0.971 — Sink Strength (base LLM) vs. IFEval Δ (VLM-LM − reference LLM). Per-head QK-RMSNorm backbones (blue) keep the sink and lose little; those without (orange) lose much more, and a layerwise backbone (green) is worst.

VLM-LM – reference LLM pair	Sink protection	Δ IFEval
InternVL3.5	per-head QK-RMSNorm	−1.8
Qwen3-VL	per-head QK-RMSNorm	−5.4
LLaVA-OneVision	none	−7.9
InternVL3	none	−8.9
Qwen2.5-VL	none	−9.6
Molmo2-O	layerwise	−18.7

Simple interventions do not close the gap: post-pretraining QK-RMSNorm injection followed by VL training does not reproduce the native protective regime, and the tested post-VL weight-merging settings do not recover the lost instruction-following ability. This points the search toward training-time, head-selective protection.

Reproduce in one shot

Tables 4–6, 8, and 10–12, the Appendix D.2 strong-sink check, and both predictor panels run offline from committed measurements. No GPU, model download, or benchmark download is needed.

# clone and reproduce the numerical results
git clone https://github.com/minsik-choi126/sink-strength
cd sink-strength
./reproduce.sh

# expected checks
#  6-pair diagnostic IFEval rho = 0.971; LOO MAE = 2.54 pt
# 17-pair IFEval rho = 0.877
#  9-pair task rhos  = 0.945 / 0.877 / 0.860 / 0.821
# strong sink (G_base > 2) = 8.5% Molmo2-O / 58% Qwen3-8B

The command validates each table row against its linked calibration JSON and regenerates the tables and machine-readable summary in outputs/, using only the Python standard library. Install the lightweight plotting requirements to regenerate the figures as well. Measuring Sink Strength from weights, extracting all 17 VLM language backbones, and rerunning the nine-task evaluation are documented in the repository README.

Code & data release

One-click paper checks. A standard-library statistics implementation rebuilds Tables 4–6, 8, and 10–12 and verifies the reported correlations, held-out errors, and D.2 strong-sink prevalence.
Released measurements. The canonical CSV links all 17 final-paper pairs to aggregate and per-head calibration JSONs, with exact Hugging Face model IDs and task deltas.
End-to-end research path. The repository includes the 15 calibration prompts, Sink Strength measurement, a 17-pair extraction registry, dense and MoE backbone converters, the nine-task text-evaluation protocol, and mechanism ablations.
Release boundary. Model weights, benchmark copies, generated checkpoints, raw evaluation samples, and internal review material are not redistributed; upstream licenses apply.

BibTeX

@inproceedings{choi2026sinkstrength,
  title   = {Text Capability Loss in Vision-Language Adaptation:
             An Attention-Sink Diagnosis},
  author  = {Choi, Minsik and Kim, Geewook and Kim, Young Geun},
  booktitle = {Proceedings of the 2026 Conference on Empirical Methods
               in Natural Language Processing},
  year    = {2026}
}