LLM Integration System

Relevant source files

• api/apps/llm_app.py
• api/db/init_data.py
• api/db/services/llm_service.py
• conf/llm_factories.json
• internal/cli/response.go
• internal/dao/tenant.go
• internal/entity/models/factory.go
• internal/entity/types.go
• internal/handler/providers.go
• internal/handler/providers_test.go
• internal/handler/tenant.go
• internal/service/generator.go
• internal/service/model_service.go
• internal/service/model_service_test.go
• internal/service/tenant.go
• internal/service/tenant_test.go
• rag/llm/__init__.py
• rag/llm/chat_model.py
• rag/llm/cv_model.py
• rag/llm/embedding_model.py
• rag/llm/rerank_model.py
• rag/llm/sequence2txt_model.py
• rag/llm/tts_model.py
• web/src/components/svg-icon.tsx
• web/src/constants/llm.ts
• web/src/utils/common-util.ts

The LLM Integration System provides a unified abstraction layer for interacting with multiple commercial and open-source AI providers. It manages tenant-specific configurations, model selection, and implements cross-cutting concerns including error handling, retry logic, usage tracking, and model-specific behavior policies. The system supports seven model types: chat, embedding, rerank, vision (image-to-text), text-to-speech (TTS), speech-to-text (ASR), and OCR.

For details on the specific components of this system, see the child pages:

• LLMBundle and Model Types — Document the LLMBundle abstraction, model type system (CHAT, EMBEDDING, RERANK, TTS, etc.), and factory pattern implementation.
• Provider Implementations — Detail the concrete provider implementations (OpenAI, Anthropic, DeepSeek, local models) and how they register with the factory, including Go-side model definitions.
• Error Handling and Retry Logic — Explain the sophisticated error classification system, retry strategies with exponential backoff, and error categorization.
• Tenant Configuration and Model Management — Document how tenants configure their own models, the relationship between tenant_llm table and model selection, and the initialization process.
• Tool Calling and Function Use — Explain the tool binding mechanism, function schema generation, and how LLMs invoke external tools during chat completion.

---

Architecture Overview

The LLM integration system follows a layered architecture that bridges high-level tenant requests to low-level provider APIs. It includes a Python-based core for RAG logic and a Go-based service layer for administrative and high-performance model management.

System Architecture: LLM Integration Layers

Architecture Description: Configuration originates from conf/llm_factories.json conf/llm_factories.json1-2 The API Layer in api/apps/llm_app.py handles tenant model setup, including verifying API keys through live tests for embedding api/apps/llm_app.py97-109 chat api/apps/llm_app.py110-131 and rerank models api/apps/llm_app.py132-147 The Service Layer uses the LLMBundle class api/db/services/llm_service.py37 to wrap model instances and handle Langfuse observation api/db/services/llm_service.py41-57 Model Registries are dynamically populated in rag/llm/__init__.py rag/llm/__init__.py145-152 by scanning provider modules listed in MODULE_MAPPING rag/llm/__init__.py155-164 The Go-side implementation uses ModelFactory internal/entity/models/factory.go24 to instantiate drivers for various providers like Anthropic, DeepSeek, and OpenAI internal/entity/models/factory.go33-165

Sources: api/apps/llm_app.py47-147 rag/llm/__init__.py145-176 rag/llm/chat_model.py44 rag/llm/embedding_model.py145 rag/llm/rerank_model.py32 conf/llm_factories.json1-2 api/db/services/llm_service.py33-57 internal/entity/models/factory.go24-165 internal/service/model_service.go130-150

---

Model Type System

RAGFlow categorizes AI capabilities into specific types, each defined by a base interface and a registry.

Model Type	Registry (Python)	Base Class	Purpose
CHAT	ChatModel	rag/llm/chat_model.py	Text generation and tool use rag/llm/chat_model.py44
EMBEDDING	EmbeddingModel	rag/llm/embedding_model.py	Vector encoding for search rag/llm/embedding_model.py145
RERANK	RerankModel	rag/llm/rerank_model.py	Scoring document relevance rag/llm/rerank_model.py32
VISION	CvModel	rag/llm/cv_model.py	Vision and image description rag/llm/cv_model.py59
TTS	TTSModel	rag/llm/tts_model.py	Text-to-speech synthesis rag/llm/tts_model.py69
ASR	Seq2txtModel	rag/llm/sequence2txt_model.py	Audio transcription rag/llm/sequence2txt_model.py35
OCR	OcrModel	rag/llm/__init__.py	Optical character recognition rag/llm/__init__.py151

For details on the abstraction and factory pattern, see LLMBundle and Model Types.

Sources: rag/llm/__init__.py145-152 rag/llm/chat_model.py44 rag/llm/embedding_model.py145 rag/llm/rerank_model.py32 rag/llm/cv_model.py59 rag/llm/tts_model.py69 rag/llm/sequence2txt_model.py35

---

Provider Implementations

The system supports a wide range of providers, from commercial APIs (OpenAI, Anthropic, DeepSeek) to local deployments (Ollama, Xinference).

Provider Registration Sequence (Python)

Registration is performed dynamically at module load time by inspecting module members for classes that define a _FACTORY_NAME rag/llm/__init__.py168-181 Providers define this name (e.g., "Jina" rag/llm/rerank_model.py97) which the system uses for routing. Supported LiteLLM providers are enumerated in SupportedLiteLLMProvider rag/llm/__init__.py25-65 In the Go backend, the ModelFactory creates specific drivers based on the provider string internal/entity/models/factory.go33-165

For details on concrete implementations, see Provider Implementations.

Sources: rag/llm/__init__.py25-65 rag/llm/__init__.py168-181 rag/llm/rerank_model.py97 internal/entity/models/factory.go33-165

---

Error Handling and Retries

RAGFlow implements a robust error classification and retry system to handle transient API failures.

• Error Classification: The system uses LLMErrorCode enums such as RATE_LIMIT_EXCEEDED, AUTH_ERROR, or QUOTA_EXCEEDED to categorize failures rag/llm/chat_model.py44-56
• Retry Logic: Concrete model implementations (e.g. CvModel) initialize retry parameters from environment variables like LLM_MAX_RETRIES and LLM_BASE_DELAY rag/llm/cv_model.py62-63
• Model Policies: Specific model families (e.g., Qwen3, Kimi-k2.5) have specialized parameter handling via _apply_model_family_policies to manage features like "thinking" or temperature constraints rag/llm/chat_model.py112-191

For details on error categorization and retry strategies, see Error Handling and Retry Logic.

Sources: rag/llm/chat_model.py44-56 rag/llm/cv_model.py62-63 rag/llm/chat_model.py112-191

---

Tenant and Model Management

Tenants configure their own API keys and base URLs, which are stored in the TenantLLM table api/apps/llm_app.py27

• Validation: When a tenant adds a model via /set_api_key api/apps/llm_app.py77 the system performs live validation by attempting a small operation (e.g., mdl.encode for embedding api/apps/llm_app.py101-107 or a stream chat for chat models api/apps/llm_app.py115-131).
• Initialization: Default models for superusers are initialized during system setup in init_superuser api/db/init_data.py47-95 ensuring the environment is ready for use immediately after deployment.
• Go Management: The Go-based ModelProviderService manages tenant model providers, instances, and groups internal/service/model_service.go142-150 supporting composite model names in format model@instance@provider internal/service/model_service.go38-67

For details on how tenants manage their models, see Tenant Configuration and Model Management.

Sources: api/apps/llm_app.py77-156 api/db/init_data.py47-95 internal/service/model_service.go38-150

---

Tool Calling

Chat models that support tool calling (indicated by is_tools: true in llm_factories.json conf/llm_factories.json19) can invoke external functions.

• State Management: The CvModel base class maintains toolcall_sessions rag/llm/cv_model.py67 to track active function calling contexts and tool definitions rag/llm/cv_model.py66
• Tool Binding: LLMBundle.bind_tools delegates the registration of tools to the underlying model implementation api/db/services/llm_service.py98-102
• Tool Resolution: The system checks if a model supports tools by resolving its is_tools attribute from the database or factory configuration api/apps/llm_app.py30-44

For details on function schema and execution, see Tool Calling and Function Use.

Sources: conf/llm_factories.json19 rag/llm/cv_model.py66-67 api/db/services/llm_service.py98-102 api/apps/llm_app.py30-44