This glossary defines codebase-specific terms, jargon, and domain concepts used throughout the Hugging Face transformers library. It provides a mapping between high-level concepts and their technical implementations for onboarding engineers.
A unified interface that allows users to instantiate the correct model, configuration, or tokenizer class from a model identifier (local path or Hub ID). It relies on mapping dictionaries that associate model types with their respective classes.
AutoModel, AutoConfig, AutoTokenizer._LazyAutoMapping src/transformers/models/auto/auto_factory.py539-550 to defer imports until a specific class is requested based on the model_type found in config.json.MODEL_MAPPING_NAMES src/transformers/models/auto/modeling_auto.py79 CONFIG_MAPPING_NAMES src/transformers/models/auto/configuration_auto.py20 and task-specific mappings like MODEL_FOR_CAUSAL_LM_MAPPING_NAMES src/transformers/models/auto/modeling_auto.py69The foundational base class for all PyTorch models in the library. It implements the complex logic for weight initialization, loading from various formats, and hardware-specific optimizations.
PreTrainedModel class src/transformers/modeling_utils.py1210-1250from_pretrained(): Handles downloading, caching, and loading state dicts src/transformers/modeling_utils.py2231-2300save_pretrained(): Serializes model weights and configuration src/transformers/modeling_utils.py2073push_to_hub(): Uploads model files to the Hugging Face Hub via PushToHubMixin src/transformers/modeling_utils.py112A high-level API that encapsulates the entire end-to-end inference process: preprocessing (tokenization/image processing), model forward pass, and postprocessing (decoding/label mapping).
Pipeline base class and the pipeline factory function src/transformers/__init__.py160-173The following diagram bridges the natural language concept of "Loading a Model" to the specific code entities involved in the from_pretrained flow, including the weight conversion system introduced in v5.
Model Loading Lifecycle
A mechanism to store the Key and Value tensors of previously processed tokens. In autoregressive decoding, this prevents recomputing the hidden states for the entire prompt at every step.
Cache base class src/transformers/cache_utils.py26 and specialized versions like DynamicCache src/transformers/cache_utils.py112 StaticCache src/transformers/cache_utils.py382 and QuantizedCache src/transformers/cache_utils.py657A pipeline of transformations applied to the raw output scores of a model. "Processors" apply constraints (e.g., forbidding specific tokens), while "Warpers" modify the distribution for sampling (e.g., temperature).
LogitsProcessorList src/transformers/generation/logits_process.py86 which chains individual processors like TemperatureLogitsWarper src/transformers/generation/logits_process.py97 or TopKLogitsWarper src/transformers/generation/logits_process.py99An optimization where a small "assistant" model generates candidate tokens quickly, which are then verified by the large "target" model in a single forward pass.
AssistedCandidateGenerator src/transformers/generation/candidate_generator.py126 and the general CandidateGenerator interface src/transformers/generation/candidate_generator.py78The primary feature-complete training loop. It handles distributed orchestration, mixed precision, and gradient accumulation.
Trainer class src/transformers/trainer.py355Accelerator, DeepSpeed, and FSDP src/transformers/trainer.py58-73The configuration schema for the Trainer. It centralizes all hyperparameters, optimization settings, and infrastructure choices.
TrainingArguments dataclass src/transformers/training_args.py180| Term | Definition | Code Pointer |
|---|---|---|
| Modular Transformers | A system for defining models as collections of reusable components to reduce code duplication. | <FileRef file-url="https://github.com/huggingface/transformers/blob/85c307a1/docs/source/en/_toctree.yml#L35-L36" min=35 max=36 file-path="docs/source/en/_toctree.yml">Hii</FileRef> |
| Weight Conversion | Infrastructure (v5) to map external weight names to Transformers naming conventions dynamically. | <FileRef file-url="https://github.com/huggingface/transformers/blob/85c307a1/src/transformers/core_model_loading.py#L48-L52" min=48 max=52 file-path="src/transformers/core_model_loading.py">Hii</FileRef> |
| SDPA | Scaled Dot Product Attention; the optimized PyTorch native attention backend. | <FileRef file-url="https://github.com/huggingface/transformers/blob/85c307a1/src/transformers/integrations/sdpa_attention.py#L1-L10" min=1 max=10 file-path="src/transformers/integrations/sdpa_attention.py">Hii</FileRef> |
| Tied Weights | Sharing parameters between the input embedding and the output LM head. | <FileRef file-url="https://github.com/huggingface/transformers/blob/85c307a1/src/transformers/modeling_utils.py#L62-L62" min=62 file-path="src/transformers/modeling_utils.py">Hii</FileRef> |
| Lazy Loading | Using _LazyModule to only import code when an attribute is accessed, speeding up library init. | <FileRef file-url="https://github.com/huggingface/transformers/blob/85c307a1/src/transformers/utils/import_utils.py#L15-L20" min=15 max=20 file-path="src/transformers/utils/import_utils.py">Hii</FileRef> |
| Safetensors | A secure, fast file format for storing tensors with zero-copy loading support. | <FileRef file-url="https://github.com/huggingface/transformers/blob/85c307a1/src/transformers/modeling_utils.py#L38-L40" min=38 max=40 file-path="src/transformers/modeling_utils.py">Hii</FileRef> |
| Flash Attention | An IO-aware exact attention algorithm for fast inference/training. | <FileRef file-url="https://github.com/huggingface/transformers/blob/85c307a1/src/transformers/integrations/flash_attention.py#L1-L10" min=1 max=10 file-path="src/transformers/integrations/flash_attention.py">Hii</FileRef> |
| Quantizer | Logic for loading and running models in reduced precision (e.g., 4-bit, 8-bit). | <FileRef file-url="https://github.com/huggingface/transformers/blob/85c307a1/src/transformers/quantizers/auto.py#L31-L31" min=31 file-path="src/transformers/quantizers/auto.py">Hii</FileRef> |
| Continuous Batching | A serving technique that schedules requests at the token level to maximize throughput. | <FileRef file-url="https://github.com/huggingface/transformers/blob/85c307a1/src/transformers/generation/continuous_batching.py#L1-L10" min=1 max=10 file-path="src/transformers/generation/continuous_batching.py">Hii</FileRef> |
| PeftAdapterMixin | A mixin class allowing models to load and manage PEFT adapters like LoRA. | <FileRef file-url="https://github.com/huggingface/transformers/blob/85c307a1/src/transformers/modeling_utils.py#L63-L63" min=63 file-path="src/transformers/modeling_utils.py">Hii</FileRef> |
| MTP | Multi-Token Prediction; a candidate generation strategy for speculative decoding. | <FileRef file-url="https://github.com/huggingface/transformers/blob/85c307a1/src/transformers/generation/candidate_generator.py#L60-L60" min=60 file-path="src/transformers/generation/candidate_generator.py">Hii</FileRef> |
This diagram maps high-level training steps to specific classes and methods that handle them within the Trainer infrastructure.
Trainer Execution Flow
Refresh this wiki
This wiki was recently refreshed. Please wait 7 days to refresh again.