Advanced Features

Relevant source files

• docs/source/en/main_classes/peft.md
• docs/source/en/peft.md
• src/transformers/conversion_mapping.py
• src/transformers/core_model_loading.py
• src/transformers/distributed/sharding_utils.py
• src/transformers/integrations/accelerate.py
• src/transformers/integrations/bitsandbytes.py
• src/transformers/integrations/eetq.py
• src/transformers/integrations/fbgemm_fp8.py
• src/transformers/integrations/mxfp4.py
• src/transformers/integrations/peft.py
• src/transformers/integrations/quanto.py
• src/transformers/models/auto/auto_factory.py
• src/transformers/models/auto/auto_mappings.py
• src/transformers/models/mra/modeling_mra.py
• src/transformers/models/yoso/modeling_yoso.py
• src/transformers/quantizers/base.py
• src/transformers/quantizers/quantizer_bnb_4bit.py
• src/transformers/quantizers/quantizer_bnb_8bit.py
• src/transformers/quantizers/quantizer_eetq.py
• src/transformers/quantizers/quantizer_fbgemm_fp8.py
• src/transformers/quantizers/quantizer_hqq.py
• src/transformers/quantizers/quantizer_mxfp4.py
• src/transformers/quantizers/quantizer_quanto.py
• src/transformers/quantizers/quantizers_utils.py
• src/transformers/utils/peft_utils.py
• tests/peft_integration/test_peft_integration.py
• tests/quantization/bnb/test_4bit.py
• tests/quantization/bnb/test_mixed_int8.py
• tests/quantization/eetq_integration/test_eetq.py
• tests/quantization/fbgemm_fp8/test_fbgemm_fp8.py
• tests/quantization/gptq/test_gptq.py
• tests/quantization/hqq/test_hqq.py
• tests/quantization/mxfp4/test_mxfp4.py
• tests/utils/test_core_model_loading.py
• tests/utils/test_distributed_sharding_utils.py
• utils/check_config_attributes.py

This page provides an overview of advanced capabilities in the Transformers library for model optimization and deployment. These features enable loading and running models efficiently in production environments through techniques like quantization, weight conversion, adapter integration, model compilation, and conversational handling.

For basic model loading and inference, see Core Architecture. For training-specific features, see Training System. For text generation capabilities, see Generation System.

Purpose and Scope

The Advanced Features subsystem encompasses six primary capability areas:

1. Quantization - Reducing model precision to save memory and accelerate inference via the HfQuantizer system.
2. Weight Conversion - Transforming checkpoint formats between different model architectures using WeightConverter and ConversionOps.
3. PEFT/Adapter Integration - Loading and managing parameter-efficient fine-tuning adapters via PeftAdapterMixin.
4. Model Export and Compilation - Optimizing models with torch.compile and static caching for production.
5. Chat Templates - Standardizing conversational inputs using Jinja2 templates and apply_chat_template.
6. Agents and Tools - Building agentic workflows with the Agent and Tool classes.

System Architecture

The following diagram illustrates how advanced features hook into the core model loading and execution flow.

Bridge: API to Code Entities

Sources: src/transformers/quantizers/base.py73-84 src/transformers/conversion_mapping.py42-141 src/transformers/integrations/peft.py57-93 src/transformers/core_model_loading.py86-105

Quantization System

The quantization system reduces model memory footprint by representing weights and activations with lower precision. The library supports a wide array of methods through the HfQuantizer base class src/transformers/quantizers/base.py73-77 and specialized implementations.

Supported Methods

The library integrates with multiple backends to provide various quantization flavors:

• bitsandbytes: 8-bit and 4-bit quantization, including NF4. Managed via Bnb4BitHfQuantizer and Bnb8BitHfQuantizer tests/quantization/bnb/test_4bit.py128-133
• MXFP4: Microscaling format support via Mxfp4Quantize, Mxfp4Dequantize, and Mxfp4Deserialize operations src/transformers/integrations/mxfp4.py49-173
• Specialized Quantizers: Support for GPTQ, AWQ, AQLM, HQQ, GGUF, TorchAO, BitNet, and FbGgemm FP8 via dedicated quantizer classes.
• Calibration: Methods like GPTQ or AWQ may require a calibration step, indicated by requires_calibration = True in the quantizer src/transformers/quantizers/base.py86

For detailed configuration and calibration details, see Quantization System.

Weight Conversion System

The weight conversion system provides a structured way to translate weights between different architectural implementations using WeightConverter src/transformers/core_model_loading.py31

Conversion Operations

The system uses ConversionOps to define transformations src/transformers/core_model_loading.py86-93 Core operations include:

• Chunk: Splitting a tensor along a dimension into equally sized chunks src/transformers/core_model_loading.py117-122
• Concatenate: Merging multiple source tensors into a single target src/transformers/core_model_loading.py141-146
• Transpose: Swapping tensor dimensions src/transformers/core_model_loading.py22
• MergeModulelist: Combining experts in MoE models into a single parameter src/transformers/core_model_loading.py27
• Renaming: Mapping old parameter names to new ones via WeightRenaming or PrefixChange src/transformers/conversion_mapping.py149-158

For implementation details and reverse conversion for saving, see Weight Conversion System.

PEFT and Adapter Integration

The library natively integrates with parameter-efficient fine-tuning (PEFT) through the PeftAdapterMixin src/transformers/integrations/peft.py57-74

Key Capabilities

• Loading: Models can load adapters through load_adapter from local paths or the Hub src/transformers/integrations/peft.py80-93
• Hotswapping: Efficiently replacing one adapter with another in-place, which avoids torch.compile recompilation src/transformers/integrations/peft.py121-127
• Management: Methods to enable_adapters(), disable_adapters(), and set_adapter() to control active weights at runtime tests/peft_integration/test_peft_integration.py165-171
• MoE Support: build_peft_weight_mapping handles complex weight mappings for Mixture-of-Expert layers.

For details on adapter state management, see PEFT and Adapter Integration.

Model Export and Compilation

For high-performance production deployment, the library provides tools to optimize the execution graph.

Production Features

• torch.compile: Integration for generating optimized kernels.
• Static Cache: Optimized KV-caching strategies for fixed-length sequence generation and graph compilation.
• Flex Attention: Support for specialized attention kernels for varied sequence lengths and sparsity.
• Export Formats: Support for ONNX and ExecuTorch for cross-platform and mobile deployment.

For optimization strategies, see Model Export and Compilation.

Chat Templates and Conversation Handling

The library handles the complexity of different chat prompt formats through the chat template system, typically defined in tokenizer_config.json.

System Components

• apply_chat_template: A tokenizer method that processes a list of messages (system, user, assistant) into a formatted string using Jinja2 templates.
• Tool Calling: Support for generating JSON schemas from Python functions for tool-use models and parsing model responses via chat_parsing_utils.py.
• System Prompts: Standardized handling of system instructions across different model families.

For template syntax and tool-calling details, see Chat Templates and Conversation Handling.

Agents and Tools System

The transformers.agents system allows models to interact with external environments by executing code or calling tools.

Bridge: Agent Logic to Code Entities

• Agent: The base class for building autonomous workflows using ReAct or code-execution strategies.
• Tool: Encapsulated functions that the agent can invoke, supporting local Python tools or remote MCP server connections.
• Production Serving: Integration with the transformers serve command to expose agentic capabilities via an OpenAI-compatible API.

For building agentic workflows, see Agents and Tools System.

Sources: src/transformers/core_model_loading.py86-181 src/transformers/conversion_mapping.py42-158 src/transformers/integrations/peft.py57-145 src/transformers/quantizers/base.py73-98 src/transformers/integrations/mxfp4.py49-176 tests/peft_integration/test_peft_integration.py90-175