This page documents the architecture and code organization of vision-language models (VLMs) in the library. It covers how pixel inputs (images and video frames) and audio signals are encoded, projected, and merged into a language model's token embedding space. The primary focus is on the Qwen family (Qwen2-VL, Qwen2.5-VL, Qwen2.5-Omni, Qwen3-VL, Qwen3-Omni-MoE), which represents the state-of-the-art in multimodal architectures with features like Multimodal RoPE (M-RoPE), windowed vision attention, and native omni-modality support. The page also details the LLaVA family, PaliGemma, Idefics, GLM4V, EXAONE 4.5, and BLIP-2.
For image preprocessing, processor classes, and input batching (pixel values, attention masks), see page 2.4. For the decoder-only backbone LLMs that these models build upon, see page 5.1. For positional encoding fundamentals including RoPE, see page 5.3.
VLMs in the library use distinct patterns for injecting multimodal information into a language model.
| Pattern | Models | Modality Merging Strategy |
|---|---|---|
| Embedding Replacement | LLaVA, PaliGemma, Qwen2/2.5/3-VL | Replace placeholder token positions (image_token_id) in the embedding sequence with projected features. |
| Cross-attention Injection | Idefics, MLLama, BLIP-2 (Q-Former) | Vision features passed as cross_attention_states or queried via a specialized transformer. |
| Mixture-of-Experts (MoE) | Qwen3-VL-MoE, Qwen3.5-MoE, GLM4V-MoE | Multimodal features routed through sparse expert layers in the backbone. |
Architecture-to-Class Bridge: Qwen & LLaVA Families
Sources: src/transformers/models/qwen2_5_vl/modeling_qwen2_5_vl.py848-950 src/transformers/models/qwen2_5_omni/modeling_qwen2_5_omni.py132-158 src/transformers/models/llava/modeling_llava.py130-137
Qwen models use a custom ViT-style encoder designed to handle both images and video at native resolution via 3D patch embedding.
| Class | Role |
|---|---|
Qwen2_5_VisionPatchEmbed | Uses nn.Conv3d with kernel [temporal_patch_size, patch_size, patch_size] to tokenize spatio-temporal volumes src/transformers/models/qwen2_5_vl/modeling_qwen2_5_vl.py99-122 |
Qwen2_5_VLVisionBlock | A transformer block inheriting from GradientCheckpointingLayer, containing VisionAttention and Qwen2_5_VLMLP src/transformers/models/qwen2_5_vl/modular_qwen2_5_vl.py144-171 |
Qwen2_5_VLPatchMerger | Merges spatial_merge_size² (usually 4) adjacent patches into a single token to reduce sequence length before the LLM src/transformers/models/qwen2_5_vl/modeling_qwen2_5_vl.py139-152 |
To manage high-resolution inputs, Qwen2.5-VL and Qwen3-VL implement Window Attention:
window_size: Defines the spatial window for local attention in most layers src/transformers/models/qwen2_5_vl/modular_qwen2_5_vl.py92fullatt_block_indexes: Specifies layers that retain global (full) attention to integrate information across windows src/transformers/models/qwen2_5_vl/modular_qwen2_5_vl.py94Multimodal RoPE (M-RoPE) assigns three position indices per token: temporal t, height h, and width w.
Qwen2VLRotaryEmbedding: Computes inverse frequencies for multimodal positions src/transformers/models/qwen2_vl/modeling_qwen2_vl.py117-135Qwen2_5_VisionRotaryEmbedding: Handles the specific rotary logic for the vision encoder tokens src/transformers/models/qwen2_5_vl/modeling_qwen2_5_vl.py125-136Qwen3_5TextRotaryEmbedding: Implements interleaved M-RoPE sections (e.g., [11, 11, 10]) to handle multidimensional spatial indexing src/transformers/models/qwen3_5_moe/modeling_qwen3_5_moe.py115-156The Omni variants extend the architecture to audio and MoE structures.
chunk_and_pad_features src/transformers/models/qwen2_5_omni/modular_qwen2_5_omni.py80-112 and get_audio_cu_seqlens src/transformers/models/qwen2_5_omni/modular_qwen2_5_omni.py115-131 for sequence packing.Qwen3_5MoeForCausalLM routes multimodal tokens through sparse expert layers. It integrates with use_experts_implementation to optimize the sparse forward pass src/transformers/models/qwen3_5_moe/modeling_qwen3_5_moe.py35-46get_llm_pos_ids_for_vision calculates the 3D grid positions for vision tokens within the LLM's sequence src/transformers/models/qwen2_5_omni/modeling_qwen2_5_omni.py164-180Sources: src/transformers/models/qwen2_5_vl/modeling_qwen2_5_vl.py99-152 src/transformers/models/qwen2_vl/modeling_qwen2_vl.py117-173 src/transformers/models/qwen2_5_omni/modular_qwen2_5_omni.py80-131 src/transformers/models/qwen3_5_moe/modeling_qwen3_5_moe.py115-156
LLaVA (Large Language-and-Vision Assistant) connects a vision backbone (e.g., CLIP) to a language model (e.g., Llama) via a linear or MLP projector.
LlavaMultiModalProjector src/transformers/models/llava/modeling_llava.py87-106:
Linear → Activation → Linear.LlavaModel.get_image_features() src/transformers/models/llava/modeling_llava.py144-169:
vision_tower.vision_feature_layer) and pooling strategies (vision_feature_select_strategy).LLaVA-NeXT introduces support for variable resolutions by tiling images.
get_anyres_image_grid_shape(): Calculates the shape of the image patch grid src/transformers/models/llava_next/modeling_llava_next.py41-70unpad_image(): Removes padding from image tensors to maintain original aspect ratios src/transformers/models/llava_next/modeling_llava_next.py109-145Sources: src/transformers/models/llava/modeling_llava.py87-106 src/transformers/models/llava_next/modeling_llava_next.py41-145
PaliGemma pairs a SigLIP vision encoder with a Gemma backbone. It is notable for its use of bidirectional attention for image tokens.
PaliGemmaMultiModalProjector: A simple linear projection layer mapping vision_config.hidden_size to projection_dim src/transformers/models/paligemma/modeling_paligemma.py90-98get_placeholder_mask(): Identifies image_token_id locations to substitute with visual features and validates that token counts match feature counts src/transformers/models/paligemma/modeling_paligemma.py151-172Idefics uses a perceiver-based or linear-projection-based architecture to integrate vision.
Idefics2VisionEmbeddings: A modified SigLIP embedding layer that supports variable resolutions via patch-level attention masks and position_embedding src/transformers/models/idefics2/modeling_idefics2.py100-128Sources: src/transformers/models/paligemma/modeling_paligemma.py90-172 src/transformers/models/idefics2/modeling_idefics2.py100-128
The core logic for merging vision and text usually resides in the forward pass of the ForConditionalGeneration classes.
Data Flow: Modality Replacement
LlavaForConditionalGeneration: Performs index-based replacement in inputs_embeds. It validates that the number of image tokens matches the features src/transformers/models/llava/modeling_llava.py151-172Qwen2_5_VLForConditionalGeneration: Uses image_grid_thw and video_grid_thw to accurately map 3D visual tokens into the text sequence src/transformers/models/qwen2_5_vl/modeling_qwen2_5_vl.py144-171Blip2ForConditionalGeneration: Uses a Q-Former (Querying Transformer) to extract a fixed number of learned queries from the vision encoder output src/transformers/models/blip_2/modeling_blip_2.py79-98Sources: src/transformers/models/llava/modeling_llava.py151-172 src/transformers/models/qwen2_5_vl/modeling_qwen2_5_vl.py144-171 src/transformers/models/blip_2/modeling_blip_2.py79-98
Multimodal models return specialized dataclasses to hold hidden states and attentions from all constituent towers.
| Dataclass | Model Family | Unique Fields |
|---|---|---|
Qwen2VLModelOutputWithPast | Qwen2-VL | rope_deltas src/transformers/models/qwen2_vl/modeling_qwen2_vl.py72-79 |
LlavaModelOutputWithPast | LLaVA | image_hidden_states src/transformers/models/llava/modeling_llava.py42-54 |
PaligemmaModelOutputWithPast | PaliGemma | image_hidden_states src/transformers/models/paligemma/modeling_paligemma.py50-58 |
Idefics2BaseModelOutputWithPast | Idefics2 | image_hidden_states (tuple) src/transformers/models/idefics2/modeling_idefics2.py48-65 |
Blip2ForConditionalGenerationModelOutput | BLIP-2 | qformer_outputs, language_model_outputs src/transformers/models/blip_2/modeling_blip_2.py79-97 |
Sources: src/transformers/models/qwen2_vl/modeling_qwen2_vl.py72-79 src/transformers/models/llava/modeling_llava.py42-54 src/transformers/models/paligemma/modeling_paligemma.py50-58 src/transformers/models/idefics2/modeling_idefics2.py48-65 src/transformers/models/blip_2/modeling_blip_2.py79-97
Refresh this wiki
This wiki was recently refreshed. Please wait 7 days to refresh again.