Overview

Relevant source files

• MIGRATION_GUIDE_V5.md
• README.md
• docs/source/en/_toctree.yml
• docs/source/en/index.md
• i18n/README_ar.md
• i18n/README_de.md
• i18n/README_es.md
• i18n/README_fr.md
• i18n/README_hd.md
• i18n/README_ja.md
• i18n/README_pt-br.md
• i18n/README_ru.md
• i18n/README_te.md
• i18n/README_ur.md
• i18n/README_vi.md
• i18n/README_zh-hans.md
• i18n/README_zh-hant.md
• pyproject.toml
• src/transformers/__init__.py
• src/transformers/_typing.py
• src/transformers/dynamic_module_utils.py
• src/transformers/image_processing_backends.py
• src/transformers/modelcard.py
• src/transformers/models/__init__.py
• src/transformers/models/auto/configuration_auto.py
• src/transformers/models/auto/feature_extraction_auto.py
• src/transformers/models/auto/image_processing_auto.py
• src/transformers/models/auto/modeling_auto.py
• src/transformers/models/auto/processing_auto.py
• src/transformers/models/auto/tokenization_auto.py
• src/transformers/models/auto/video_processing_auto.py
• src/transformers/models/chameleon/image_processing_chameleon.py
• src/transformers/models/cohere/tokenization_cohere.py
• src/transformers/models/flava/image_processing_pil_flava.py
• src/transformers/models/nllb/tokenization_nllb.py
• src/transformers/models/vits/modeling_vits.py
• src/transformers/pipelines/__init__.py
• src/transformers/pipelines/base.py
• src/transformers/utils/dummy_pt_objects.py
• tests/models/albert/test_tokenization_albert.py
• tests/models/auto/test_configuration_auto.py
• tests/models/auto/test_feature_extraction_auto.py
• tests/models/auto/test_image_processing_auto.py
• tests/models/auto/test_modeling_auto.py
• tests/models/auto/test_video_processing_auto.py
• tests/models/barthez/test_tokenization_barthez.py
• tests/models/bert_generation/test_tokenization_bert_generation.py
• tests/models/big_bird/test_tokenization_big_bird.py
• tests/models/biogpt/test_tokenization_biogpt.py
• tests/models/cpmant/test_modeling_cpmant.py
• tests/models/cpmant/test_tokenization_cpmant.py
• tests/models/herbert/test_tokenization_herbert.py
• tests/models/mbart/test_tokenization_mbart.py
• tests/models/nllb/test_tokenization_nllb.py
• tests/models/pegasus/test_tokenization_pegasus.py
• tests/models/plbart/test_tokenization_plbart.py
• tests/models/seamless_m4t/test_tokenization_seamless_m4t.py
• tests/models/vits/test_modeling_vits.py
• tests/pipelines/test_pipelines_common.py
• tests/repo_utils/test_check_auto.py
• tests/repo_utils/test_checkers.py
• tests/utils/test_dynamic_module_utils.py
• utils/check_repo.py
• utils/check_types.py
• utils/checkers.py

Transformers is a model-definition framework for state-of-the-art machine learning across text, computer vision, audio, video, and multimodal domains. It provides standardized implementations of model architectures for both inference and training, serving as the central pivot between the Hugging Face Hub (1M+ model checkpoints), training frameworks (Axolotl, Unsloth, DeepSpeed, FSDP), inference engines (vLLM, SGLang, TGI), and adjacent modeling libraries (llama.cpp, mlx). src/transformers/models/auto/modeling_auto.py41-490 src/transformers/__init__.py21-22 README.md69-75

Scope of this document: This page provides a high-level overview of the Transformers library's purpose, architecture, and capabilities. For detailed information about specific systems, see: Core Architecture for model loading and tokenization, Training System for fine-tuning capabilities, Generation System for text generation, Model Architectures for specific model families, and Advanced Features for quantization and optimization.

Purpose and Core Philosophy

Transformers centralizes model definitions to ensure compatibility across the machine learning ecosystem. A model implemented in Transformers works seamlessly with multiple frameworks and tools without requiring reimplementation. README.md72-75 docs/source/en/index.md25-28

Core design principles:

1. Fast and easy to use: Every model implementation is built around three main classes: configuration (PreTrainedConfig), model (PreTrainedModel), and preprocessor (such as PreTrainedTokenizerBase or ProcessorMixin). docs/source/en/index.md54 src/transformers/configuration_utils.py66 src/transformers/tokenization_utils_base.py190 src/transformers/processing_utils.py178
2. Pretrained models: Reduce compute cost and carbon footprint by leveraging pre-existing checkpoints reproduced as closely as possible to original implementations. docs/source/en/index.md55
3. Ecosystem compatibility: Acts as the model-definition framework for the majority of training frameworks (DeepSpeed, FSDP) and inference engines (vLLM, TGI). README.md73-75 docs/source/en/index.md26-28

Sources: README.md69-82 docs/source/en/index.md22-31 src/transformers/configuration_utils.py66 src/transformers/processing_utils.py178

Library Architecture

The library is organized into several interconnected subsystems that handle the complete model lifecycle from loading to inference to training.

High-Level System Organization

Sources: src/transformers/__init__.py63-274 src/transformers/models/auto/configuration_auto.py36-48 src/transformers/models/auto/modeling_auto.py41-490 src/transformers/models/auto/tokenization_auto.py65-150

Core Capabilities

1. Model Loading and Inference

The AutoModel system provides automatic model class selection based on the configuration of the checkpoint. src/transformers/models/auto/modeling_auto.py41-490

Component	Purpose	Key Classes
Auto Config	Load configuration	AutoConfig src/transformers/models/auto/configuration_auto.py42
Auto Model	Instantiate model	AutoModel, AutoModelForCausalLM src/transformers/models/auto/modeling_auto.py23 src/transformers/models/auto/modeling_auto.py41
Auto Tokenizer	Load tokenizer	AutoTokenizer src/transformers/models/auto/tokenization_auto.py14
Auto Processor	Load multimodal processor	AutoProcessor src/transformers/models/auto/processing_auto.py14

Example workflow:

Sources: src/transformers/models/auto/modeling_auto.py1-490 src/transformers/models/auto/configuration_auto.py1-144 src/transformers/models/auto/tokenization_auto.py1-150

2. High-Level Inference with Pipeline

The Pipeline API provides task-oriented inference. It abstracts away the complexity of preprocessing, model execution, and postprocessing. src/transformers/pipelines/base.py124-126 README.md124-128

Supported tasks include: text-generation, automatic-speech-recognition, image-classification, object-detection, visual-question-answering, and many more across modalities. src/transformers/__init__.py141-173 docs/source/en/index.md43

Sources: src/transformers/__init__.py141-173 src/transformers/pipelines/base.py1-184 README.md124-130

3. Training and Fine-tuning

The library provides the Trainer API, a comprehensive training loop for PyTorch that supports features like mixed precision and distributed training. docs/source/en/index.md44 src/transformers/__init__.py195-200

Sources: src/transformers/__init__.py195-200 docs/source/en/index.md44 docs/source/en/_toctree.yml179-183

4. Text Generation

The generate() API supports fast text generation for LLMs and VLMs, including streaming and multiple decoding strategies (greedy, sampling, beam search). docs/source/en/index.md45 src/transformers/__init__.py113-122

Sources: src/transformers/__init__.py113-122 docs/source/en/index.md45 docs/source/en/_toctree.yml92-97

Model Lifecycle: Hub to Deployment

The following diagram illustrates how models flow from the Hugging Face Hub through the Transformers library to various deployment targets:

Sources: src/transformers/models/auto/modeling_auto.py41-490 src/transformers/models/auto/configuration_auto.py103-118 README.md73-75 docs/source/en/index.md26-28

Key Abstractions and Design Patterns

Auto Classes and Lazy Loading

The library uses a lazy loading mechanism via _LazyModule and _LazyAutoMapping to defer imports until objects are requested. This keeps the initial import of the library fast. src/transformers/__init__.py15-19 src/transformers/models/auto/auto_factory.py24

Mapping Dictionary	Maps To	Example Entry
CONFIG_MAPPING_NAMES	Configuration classes	("bert", "BertConfig") src/transformers/models/auto/configuration_auto.py36-48
MODEL_MAPPING_NAMES	Base model classes	("llama", "LlamaModel") src/transformers/models/auto/modeling_auto.py186
TOKENIZER_MAPPING_NAMES	Tokenizer backends	("bert", "BertTokenizer") src/transformers/models/auto/tokenization_auto.py78

Sources: src/transformers/__init__.py15-19 src/transformers/models/auto/configuration_auto.py36-48 src/transformers/models/auto/modeling_auto.py41-490 src/transformers/models/auto/tokenization_auto.py65-150

Modular Processing Pipeline

Processing components handle different modalities and are unified into a BatchEncoding or BatchFeature structure for model consumption. src/transformers/tokenization_utils_base.py188 src/transformers/feature_extraction_utils.py111

Sources: src/transformers/__init__.py174-181 src/transformers/tokenization_utils_base.py186-192 src/transformers/processing_utils.py174-181

Supported Modalities and Model Families

The library supports a vast array of model families across text (BERT, Llama, Mistral), vision (ViT, ConvNeXT, DETR), audio (Whisper, Wav2Vec2), and multimodal (Qwen2-VL, Idefics) tasks. src/transformers/models/auto/modeling_auto.py41-490 src/transformers/models/auto/image_processing_auto.py72-124

Sources: src/transformers/models/auto/modeling_auto.py41-490 docs/source/en/_toctree.yml15-74

Installation and Requirements

Requirements: Python 3.10+, PyTorch 2.4+. README.md86

Sources: README.md84-120 src/transformers/__init__.py21-30