Model Architectures

Relevant source files

This page provides an overview of the major model architecture families supported by the transformers library and the common architectural patterns shared across implementations. The library supports 200+ model architectures, organized under src/transformers/models/, each implementing specific architectural innovations while sharing fundamental building blocks provided by PreTrainedModel in src/transformers/modeling_utils.py.

Architecture Families

Family	Representative Models	Child Page
Decoder-Only LMs	LLaMA family (v1-v4), Mistral, Gemma (1-4), Qwen (2-3.5), Phi, Falcon, EXAONE	Decoder-Only Language Models
Attention Mechanisms	Eager, Flash Attention 2, SDPA, FlexAttention, GQA, MQA, Sliding Window, Paged Attention	Attention Mechanisms
Positional Embeddings	RoPE (Linear, Dynamic, YaRN), MRoPE (Vision-Language), Sinusoidal, Learned	Positional Embeddings
Mixture-of-Experts	Mixtral, Qwen2MoE, Jamba, DeepSeek V2/V3/V4, SonicMoE	Mixture-of-Experts Architecture
ASR / Speech	Whisper, Bark, SpeechT5, Wav2Vec2, Qwen2.5-Omni (Audio), CSM	Whisper and Automatic Speech Recognition
Encoder-Decoder	T5, BART, mBART, Pegasus, Marian, SwitchTransformers, Pix2Struct	Encoder-Decoder Models
Multimodal VLMs	LLaVA, PaliGemma, Qwen2.5-VL, Idefics, Gemma3, Gemma4, LLaMA4 (Multimodal), GLM4V	Multimodal Vision-Language Models
Vision Models	CLIP, SigLIP, ViT, DINOv2, DETR, SAM, Swin, RT-DETR	Vision and Computer Vision Models
State Space Models	Mamba, Mamba2, RWKV, Jamba (Hybrid), Bamba, Zamba2, RecurrentGemma	State Space and Recurrent Models
Audio Generation	SpeechT5, MusicGen, Bark, Moshi, Voxtral, EnCodec, DAC	Audio and Speech Generation Models

All model families share common building blocks (embeddings, attention mechanisms, feed-forward networks, normalization layers) while varying in their specific implementations and combinations.

Related Documentation:

For model loading mechanisms, see Model Loading and Weight Management
For training models, see Training System
For text generation, see Generation System

Model Implementation Structure

All model implementations in transformers follow a consistent hierarchy that provides both flexibility and standardization across different architectures.

Model Class Hierarchy

Sources: src/transformers/models/llama/modeling_llama.py40 src/transformers/models/mistral/modeling_mistral.py26 src/transformers/models/gemma3/modeling_gemma3.py50 src/transformers/models/qwen2_vl/modeling_qwen2_vl.py45 src/transformers/modeling_layers.py30-34

Core Model Components

Modern LLM architectures consist of hierarchical components that combine into complete models. The diagram below maps the "Natural Language Space" concepts to the "Code Entity Space" using the Llama implementation as a reference.

Component Hierarchy Diagram

Sources: src/transformers/models/llama/modeling_llama.py315-600 src/transformers/models/llama/modeling_llama.py173-197 src/transformers/models/llama/modeling_llama.py132-157 src/transformers/models/llama/modeling_llama.py53-71

Model Categories and Implementation Patterns

Decoder-Only Language Models

Most modern LLMs follow the decoder-only transformer architecture. Optimized implementations exist for LLaMA (v1-v4), Mistral, Gemma (1-4), and Qwen (2-3.5) families. These models utilize Causal Language Modeling heads and advanced attention mechanisms like Grouped-Query Attention (GQA) in MistralAttention src/transformers/models/mistral/modeling_mistral.py122-138 For detailed architecture, see Decoder-Only Language Models.

Sources: src/transformers/models/llama/modeling_llama.py315-600 src/transformers/models/mistral/modeling_mistral.py122-180 src/transformers/models/gemma/modeling_gemma.py84-140

Mixture-of-Experts (MoE) Models

MoE models like Mixtral and Qwen2-Moe use sparse expert routing to scale parameters. The routing logic typically involves a gate network (e.g., MixtralTopKRouter) selecting the top-k experts src/transformers/models/mixtral/modeling_mixtral.py96-112 The MixtralExperts class stores expert weights as 3D tensors to optimize throughput src/transformers/models/mixtral/modeling_mixtral.py57-93 For detailed MoE internals, see Mixture-of-Experts Architecture.

Sources: src/transformers/models/mixtral/modeling_mixtral.py57-130 src/transformers/models/qwen2_moe/modeling_qwen2_moe.py62-146

Multimodal Vision-Language Models

Models like Gemma3, Qwen2-VL, and Qwen2.5-VL integrate vision encoders with LLM backbones. Qwen2.5-Omni extends this to audio and video, utilizing specialized chunking logic like chunk_and_pad_features src/transformers/models/qwen2_5_omni/modular_qwen2_5_omni.py80-112 Vision inputs are processed via Qwen2_5_VisionPatchEmbed src/transformers/models/qwen2_5_vl/modeling_qwen2_5_vl.py99-123 before being merged via Qwen2_5_VLPatchMerger src/transformers/models/qwen2_5_vl/modeling_qwen2_5_vl.py139-152 For details, see Multimodal Vision-Language Models.

Sources: src/transformers/models/gemma3/modeling_gemma3.py66-104 src/transformers/models/qwen2_5_vl/modeling_qwen2_5_vl.py139-152 src/transformers/models/qwen2_5_omni/modular_qwen2_5_omni.py15-62

Common Architectural Components

Attention Implementation Strategy

Models support multiple attention backends, including Eager, SDPA, and Flash Attention 2. Grouped-Query Attention (GQA) is standard in Mistral models src/transformers/models/mistral/modeling_mistral.py130-131 Advanced masking utilities in masking_utils.py support complex patterns like sliding windows src/transformers/models/mistral/modeling_mistral.py16 Qwen2.5-VL introduces window attention for high-resolution images via get_vision_window_index src/transformers/models/qwen2_5_vl/modeling_qwen2_5_vl.py57

Sources: src/transformers/models/mistral/modeling_mistral.py96-118 src/transformers/models/llama/modeling_llama.py199-295 src/transformers/models/qwen2_5_vl/modeling_qwen2_5_vl.py57

Positional Encoding Patterns

Rotary Position Embeddings (RoPE) are the dominant encoding scheme. Implementations vary from standard RoPE to hybrid configurations (Gemma3) that use different rope_theta values for different layer types (e.g., sliding_attention vs full_attention) src/transformers/models/gemma3/modular_gemma3.py135-157 Qwen2-VL uses a 3D multimodal RoPE expanded to handle grid-based vision inputs src/transformers/models/qwen2_vl/modeling_qwen2_vl.py158-171

Model Family	Implementation Class	Key Characteristic
Llama / Mistral	`LlamaRotaryEmbedding`	Standard RoPE
Qwen2-VL	`Qwen2VLRotaryEmbedding`	3D Multimodal RoPE
Gemma3	`Gemma3RotaryEmbedding`	Hybrid theta for sliding/full layers
Qwen2.5-VL	`Qwen2_5_VisionRotaryEmbedding`	3D Vision-specific RoPE

Sources: src/transformers/models/llama/modeling_llama.py73-130 src/transformers/models/qwen2_vl/modeling_qwen2_vl.py117-171 src/transformers/models/gemma3/modeling_gemma3.py156-170 src/transformers/models/qwen2_5_vl/modeling_qwen2_5_vl.py125-137

Normalization and MLP Strategies

RMSNorm: Standard in Llama, Mistral, and Mixtral. Gemma variants use a 1.0 + weight modification in GemmaRMSNorm src/transformers/models/gemma/modeling_gemma.py77-78 and Gemma3RMSNorm src/transformers/models/gemma3/modeling_gemma3.py149-150
MLP: Most models use a gated SwiGLU structure with gate_proj, up_proj, and down_proj. The act_fn is typically silu or gelu src/transformers/models/mistral/modeling_mistral.py41-44

Sources: src/transformers/models/llama/modeling_llama.py53-70 src/transformers/models/gemma/modeling_gemma.py64-81 src/transformers/models/gemma3/modeling_gemma3.py136-154 src/transformers/models/mistral/modeling_mistral.py35-49

Model Architectures

Architecture Families

Model Implementation Structure

Model Class Hierarchy

Core Model Components

Model Categories and Implementation Patterns

Decoder-Only Language Models

Mixture-of-Experts (MoE) Models

Multimodal Vision-Language Models

Common Architectural Components

Attention Implementation Strategy

Positional Encoding Patterns

Normalization and MLP Strategies

On this page

Model Architectures

Architecture Families

Model Implementation Structure

Model Class Hierarchy

Core Model Components

Model Categories and Implementation Patterns

Decoder-Only Language Models

Mixture-of-Experts (MoE) Models

Multimodal Vision-Language Models

Common Architectural Components

Attention Implementation Strategy

Positional Encoding Patterns

Normalization and MLP Strategies

On this page