This document covers the positional embedding systems used throughout the transformers library to encode sequence position information into transformer models. The primary focus is on Rotary Position Embeddings (RoPE) and its variants, which are the dominant positional encoding mechanism used in modern decoder-only language models and vision-language models.
For information about attention mechanisms that consume these embeddings, see Attention Mechanisms. For details on model architectures that use these embeddings, see Decoder-Only Language Models and Multimodal Vision-Language Models.
Positional embeddings enable transformer models to capture the sequential nature of input data. Unlike recurrent architectures, transformers process all tokens in parallel and require explicit position information. This library implements several positional encoding schemes:
Sources: src/transformers/models/llama/configuration_llama.py80 src/transformers/modeling_rope_utils.py1-130 src/transformers/models/mixtral/configuration_mixtral.py98 src/transformers/models/qwen2_5_vl/modeling_qwen2_5_vl.py124 src/transformers/models/qwen2_5_omni/modeling_qwen2_5_omni.py133-160
The core RoPE implementation follows a consistent pattern across all decoder-only models. Each model has a RotaryEmbedding class that inherits common behavior but can be customized per architecture.
The following diagram maps the code entities involved in RoPE initialization and application to their logical roles in the transformer architecture.
Title: RoPE System Entity Mapping
Sources: src/transformers/modeling_rope_utils.py33-130 src/transformers/models/llama/configuration_llama.py80 src/transformers/models/qwen2_vl/modeling_qwen2_vl.py116-130 src/transformers/models/qwen2_5_vl/modeling_qwen2_5_vl.py161-163 src/transformers/models/qwen3_5/modeling_qwen3_5.py149-153
Models define RoPE requirements in their configuration classes. Configuration keys like rope_theta define the base frequency for the inverse frequency calculations src/transformers/models/llama/configuration_llama.py80 Modern multimodal models often use mrope_section to define dimension splits for different axes (time, height, width) src/transformers/models/qwen3_5/modeling_qwen3_5.py117
Key Configuration Parameters:
rope_theta: The base frequency (e.g., 10000.0 or 1000000.0) src/transformers/models/qwen2_vl/modeling_qwen2_vl.py154 src/transformers/models/qwen3_vl/modular_qwen3_vl.py139partial_rotary_factor: Used in architectures like Phi or Qwen3.5 to apply RoPE only to a fraction of the head dimension src/transformers/models/phi/configuration_phi.py83-85 src/transformers/models/qwen3_5/modeling_qwen3_5.py138original_max_position_embeddings: Required for scaling strategies to define the pre-training context src/transformers/modeling_rope_utils.py55dynamic_rope_update DecoratorThe dynamic_rope_update decorator wraps the forward method of RotaryEmbedding classes to support dynamic context window extension src/transformers/modeling_rope_utils.py34-45
It provides two main update mechanisms:
dynamic_frequency_update: Recomputes inv_freq when the sequence length grows beyond the current cache src/transformers/modeling_rope_utils.py82-119longrope_frequency_update: Switches between short_factor and long_factor based on whether the sequence exceeds original_max_position_embeddings src/transformers/modeling_rope_utils.py47-80The forward pass computes rotary embeddings by multiplying inv_freq with position_ids. In multimodal models like Qwen2-VL or Qwen3.5, inv_freq is expanded and indexed to handle multiple position dimensions src/transformers/models/qwen2_vl/modeling_qwen2_vl.py166-170 src/transformers/models/qwen3_5/modeling_qwen3_5.py155-160
The application involves the apply_rotary_pos_emb function (or apply_rotary_pos_emb_vision for vision states), which uses a rotate_half helper to transform the query and key tensors src/transformers/models/qwen2_5_vl/modeling_qwen2_5_vl.py155-163
| Model | Implementation | Distinguishing Feature |
|---|---|---|
| Qwen2-VL | Qwen2VLRotaryEmbedding | Expands inv_freq for 3D grid position IDs src/transformers/models/qwen2_vl/modeling_qwen2_vl.py116 |
| Qwen2.5-VL | Qwen2_5_VisionRotaryEmbedding | Specialized for vision patch embeddings with 3D convolution src/transformers/models/qwen2_5_vl/modeling_qwen2_5_vl.py125 |
| Qwen3.5 | Qwen3_5TextRotaryEmbedding | Uses dynamic_rope_update and handles mrope_section src/transformers/models/qwen3_5/modeling_qwen3_5.py99 |
| GLM-4V | Glm4vVisionModel | Uses Qwen2_5_VisionRotaryEmbedding for its vision backbone src/transformers/models/glm4v/modular_glm4v.py53 |
Sources: src/transformers/modeling_rope_utils.py34-130 src/transformers/models/qwen2_vl/modeling_qwen2_vl.py116-170 src/transformers/models/qwen2_5_vl/modeling_qwen2_5_vl.py125-163 src/transformers/models/qwen3_5/modeling_qwen3_5.py99-165
The library supports multiple scaling strategies via ROPE_INIT_FUNCTIONS defined in modeling_rope_utils.py src/transformers/modeling_rope_utils.py132-138
| Strategy | Implementation Function | Logic |
|---|---|---|
| Linear | _compute_linear_scaling_rope_parameters | Scales frequencies by a constant factor src/transformers/modeling_rope_utils.py133-138 |
| Dynamic NTK | _compute_dynamic_ntk_parameters | Dynamically recomputes NTK-aware frequencies as sequence grows src/transformers/modeling_rope_utils.py200-240 |
| YaRN | _compute_yarn_parameters | Combines linear scaling with a temperature parameter src/transformers/modeling_rope_utils.py243-300 |
| LongRoPE | longrope_frequency_update | Uses pre-defined long/short factors src/transformers/modeling_rope_utils.py47-80 |
Sources: src/transformers/modeling_rope_utils.py40-300
For vision-language models, RoPE is extended to handle multi-dimensional inputs (e.g., temporal, height, width). This is managed by providing specific rope_parameters in the model's configuration, often using a mrope_section to define the dimensions for each modality tests/models/qwen2_5_vl/test_modeling_qwen2_5_vl.py134
Title: MRoPE Positional Embedding Calculation Flow
Sources: src/transformers/modeling_rope_utils.py34-119 src/transformers/models/qwen2_vl/modeling_qwen2_vl.py166-170 src/transformers/models/qwen2_5_vl/modeling_qwen2_5_vl.py161-173 tests/models/qwen2_5_vl/test_modeling_qwen2_5_vl.py134 src/transformers/models/qwen3_5/modeling_qwen3_5.py155-160
While RoPE is the standard for attention-based positioning, some components use fixed sinusoidal embeddings.
Qwen2_5Omni use SinusoidsPositionEmbedding for singular positional encoding in audio processing src/transformers/models/qwen2_5_omni/modeling_qwen2_5_omni.py146-148compute_default_singular_positional_embedding src/transformers/models/qwen2_5_omni/modeling_qwen2_5_omni.py148Sources: src/transformers/models/qwen2_5_omni/modeling_qwen2_5_omni.py133-160
Several architectures (Mistral, Mixtral, Gemma2) utilize sliding window attention (SWA) to manage long sequences by restricting the attention span.
sliding_window attribute in the config src/transformers/models/mixtral/configuration_mixtral.py98 src/transformers/models/gemma2/configuration_gemma2.py109sliding_attention and full_attention src/transformers/models/gemma2/configuration_gemma2.py116-119create_sliding_window_causal_mask to enforce the window constraint during the forward pass src/transformers/models/qwen2_vl/modeling_qwen2_vl.py36Sources: src/transformers/models/mixtral/configuration_mixtral.py98 src/transformers/models/gemma2/configuration_gemma2.py109-119 src/transformers/models/qwen2_vl/modeling_qwen2_vl.py36
Refresh this wiki
This wiki was recently refreshed. Please wait 7 days to refresh again.