This page documents the persistent memory subsystem in RAGFlow. The memory system provides agents with long-term context by categorizing interaction data into specific types, extracting structured information using LLMs, and storing them in specialized document store indices with per-user isolation.
RAGFlow implements four distinct memory types to handle different aspects of agent-user interactions. While "Raw" memory is a mandatory baseline, the other three types use LLMs to analyze and structure dialogue for better retrieval.
| Memory Type | Description | Implementation Detail |
|---|---|---|
| Raw | The literal transcript of the conversation between the user and the agent. | Stored as baseline dialogue content api/db/joint_services/memory_message_service.py68-83 |
| Semantic | Universal facts, definitions, concepts, and relationships. | Time-invariant information extracted via LLM internal/service/memory.go124-132 |
| Episodic | Specific experiences, events, and personal stories. | Time-bound, person-specific contextual records internal/service/memory.go133-141 |
| Procedural | Processes, methods, and step-by-step instructions. | Goal-oriented actionable knowledge internal/service/memory.go142-150 |
The Go-based PromptAssembler handles the assembly of system prompts for these extractions, using the SYSTEM_BASE_TEMPLATE internal/service/memory.go107-120 and specific instructions from TYPE_INSTRUCTIONS internal/service/memory.go123-151 In the Python layer, the MemoryType enum and bitwise operations are used to manage these types api/db/services/memory_service.py97-99
Sources: internal/service/memory.go102-188 api/db/joint_services/memory_message_service.py68-83 api/db/services/memory_service.py97-99
RAGFlow uses a polyglot persistence strategy. Metadata such as configurations and memory definitions are managed via the database service layer (MySQL/PostgreSQL) using MemoryService api/db/services/memory_service.py28-30 while high-volume message content and vectors are handled by specialized document store connectors via MessageService memory/services/messages.py34-35
Messages are stored in indices partitioned by user/tenant ID to ensure data isolation. The naming convention for these indices is generated via index_name(uid), which results in memory_{prefix}_{uid} memory/services/messages.py29-31
For storage engines like Elasticsearch, the system ensures specific vector mappings for these memory indices. The ESConnection class handles the mapping of message dictionaries to storage-ready documents memory/utils/es_conn.py52-77
The following diagram illustrates the flow from a user interaction to persistent storage, including the LLM extraction step for non-raw memory types.
Sequence: Message Extraction and Persistence
Sources: api/apps/services/memory_api_service.py75-113 api/db/joint_services/memory_message_service.py39-102 memory/services/messages.py51-54
This diagram bridges the conceptual memory space with the internal classes and storage connectors.
Memory Entity Mapping
Sources: api/db/services/memory_service.py28-30 memory/services/messages.py34-35 internal/service/memory.go17-35 memory/utils/es_conn.py36-37 common/doc_store/ob_conn_base.py34-42
The MessageService provides a unified interface for managing message indices and documents across different storage backends (Elasticsearch, Infinity, OceanBase).
page_size is 50 memory/services/messages.py69The system supports complex search and retrieval operations:
ESConnection.search supports FusionExpr with weighted_sum to combine text and vector scores memory/utils/es_conn.py159-170map_message_to_es_fields transforms internal message dictionaries into storage-ready documents, including tokenization and dynamic vector field naming based on dimensions (e.g., q_1536_vec) memory/utils/es_conn.py52-77get_recent_messages retrieves context for the current session, ordered by valid_at memory/services/messages.py125-148Sources: internal/service/memory.go38-66 memory/services/messages.py69-122 memory/utils/es_conn.py52-170
Memory entries and messages are stored with consistent schemas defined in mapping files and connection utilities.
Elasticsearch indices for memory use dynamic templates to handle various field types automatically, such as *_ltks for tokenized text and *_vec for vectors conf/mapping.json83-202
| Field Name | Storage Type | Purpose |
|---|---|---|
message_id | long | Unique ID for the message entry memory/utils/es_conn.py61 |
message_type_kwd | keyword | Categorization (raw, semantic, etc.) memory/utils/es_conn.py62 |
status_int | integer | Active/Inactive status memory/utils/es_conn.py71 |
content_ltks | text | Original message content memory/utils/es_conn.py73 |
tokenized_content_ltks | text | Fine-grained tokenized content for search memory/utils/es_conn.py74 |
q_{dims}_vec | dense_vector | Embedding vector for semantic search memory/utils/es_conn.py75 |
Access to memory resources is governed by TenantPermission levels defined in both Python and Go layers:
Sources: conf/mapping.json25-210 memory/utils/es_conn.py52-77 internal/service/memory.go50-60
Refresh this wiki
This wiki was recently refreshed. Please wait 4 days to refresh again.