Questions about architecture recipe

[issunsun1]

1. Why use the concatenated multi-layer features from the visual encoder as the KV for the text query?
2. Do the selected specific layer visual features have any special effects or basis?

Thank you!

[Jihuai-wpy]

Thanks for the questions!

1. Why concatenate multi-layer ViT features as the K/V for the text queries?

Two reasons.

The first is lineage. Our vision encoder is initialized from the Qwen3-VL ViT, which was pre-trained with DeepStack, i.e. features are tapped at a fixed set of intermediate layers and fed to the language model. We keep the same tap points so that this already-aligned information is used rather than discarded by keeping only the last layer. Our motivation is the same as in DeepStack and Qwen3-VL: the final ViT layer is the most semantic and tends to wash out low-level details such as small text, edges, and precise locations, while mixing in shallower layers preserves multi-scale information that helps OCR, grounding, and fine-grained perception.

The second is why we concatenate instead of the layer-wise injection used by DeepStack. Qwen3-VL adds the features from different intermediate layers to the hidden states at the visual-token positions in the first few layers of the language model. MOSS-VL is different: visual K/V enters only through the 12 gated cross-attention layers. If we followed the layer-wise scheme and assigned each feature level to a different cross-attention layer, every layer could attend to only a single level, which is too little information; moreover, deciding in advance which layer sees which level introduces a prior bias. So we concatenate the four levels and feed them to every cross-attention layer at once, letting the model decide which visual information it needs at each layer.

2. Is there anything special about the selected layers?

The taps are after blocks 8, 16, and 24 of the 27-layer ViT plus the final output, i.e. roughly evenly spaced through the depth; they correspond to deepstack_visual_indexes in config.json. These indices are identical to Qwen3-VL's, and we keep them so that the initialization stays consistent with the tap points the ViT was pre-trained with. The projection module is newly initialized and trained in Stage 1 (Sec. 3.1 of the report); only the tap positions are inherited. One design detail worth noting: each of the four levels has its own LayerNorm with no parameter sharing, because the scale of shallow-layer features differs from that of the final layer.

[issunsun1]

Thank you very much for your thoughtful and detailed answer. This was very helpful :)