Data Storage Architecture

Relevant source files

• api/apps/restful_apis/chunk_api.py
• api/apps/restful_apis/dataset_api.py
• api/apps/services/dataset_api_service.py
• api/db/joint_services/user_account_service.py
• api/db/services/doc_metadata_service.py
• common/asyncio_utils.py
• common/data_source/blob_connector.py
• common/data_source/confluence_connector.py
• common/data_source/discord_connector.py
• common/data_source/dropbox_connector.py
• common/data_source/jira/connector.py
• common/data_source/models.py
• common/data_source/notion_connector.py
• common/data_source/slack_connector.py
• common/data_source/utils.py
• common/doc_store/doc_store_base.py
• common/doc_store/es_conn_base.py
• common/doc_store/infinity_conn_base.py
• common/doc_store/infinity_conn_pool.py
• common/doc_store/ob_conn_base.py
• common/doc_store/ob_conn_pool.py
• common/http_client.py
• common/metadata_utils.py
• common/settings.py
• conf/infinity_mapping.json
• memory/utils/aggregation_utils.py
• memory/utils/highlight_utils.py
• memory/utils/ob_conn.py
• rag/advanced_rag/knowlege_compile/_common.py
• rag/advanced_rag/knowlege_compile/dataset_nav.py
• rag/advanced_rag/knowlege_compile/raptor.py
• rag/advanced_rag/knowlege_compile/runner.py
• rag/advanced_rag/knowlege_compile/structure.py
• rag/advanced_rag/knowlege_compile/wiki.py
• rag/flow/compiler/compiler.py
• rag/graphrag/general/smoke.py
• rag/graphrag/light/smoke.py
• rag/graphrag/search.py
• rag/graphrag/utils.py
• rag/svr/cache_file_svr.py
• rag/svr/task_executor_refactor/chunk_post_processor.py
• rag/svr/task_executor_refactor/dataflow_service.py
• rag/svr/task_executor_refactor/dataset_structure_merger.py
• rag/svr/task_executor_refactor/dataset_wiki_generator.py
• rag/svr/task_executor_refactor/task_handler.py
• rag/utils/azure_sas_conn.py
• rag/utils/azure_spn_conn.py
• rag/utils/es_conn.py
• rag/utils/infinity_conn.py
• rag/utils/minio_conn.py
• rag/utils/ob_conn.py
• rag/utils/opendal_conn.py
• rag/utils/opensearch_conn.py
• rag/utils/oss_conn.py
• rag/utils/s3_conn.py
• test/testcases/test_web_api/test_chunk_app/test_chunk_routes_unit.py
• test/unit_test/common/test_dropbox_connector.py
• test/unit_test/common/test_metadata_filter_operators.py
• test/unit_test/data_source/test_slack_connector_unit.py
• test/unit_test/memory/utils/test_ob_conn_aggregation.py
• test/unit_test/memory/utils/test_ob_conn_highlight.py
• test/unit_test/rag/graphrag/test_graphrag_utils.py
• test/unit_test/rag/test_raptor_psi_tree_builder.py
• test/unit_test/rag/utils/test_opensearch_doc_meta.py
• web/src/assets/svg/data-source/slack.svg
• web/src/components/ui/tree-view.tsx

Purpose and Scope

This document details the polyglot persistence strategy employed by RAGFlow. The system integrates multiple storage technologies—MySQL/PostgreSQL as relational databases, document stores (Elasticsearch, Infinity, OceanBase, or OpenSearch), object storage (MinIO/S3/OSS via OpenDAL abstraction), and Redis for caching and coordination. It covers the implementation details, data flow, key classes and functions, and how these components interoperate to support retrieval-augmented generation (RAG) workflows.

---

Storage Layer Overview

RAGFlow's storage architecture is designed as a multi-tier system that uses different storage engines optimized for specific tasks:

• Relational Databases (MySQL/PostgreSQL): Store structured metadata and user/tenant data with ACID guarantees.
• Document Stores (Elasticsearch/Infinity/OceanBase/OpenSearch): Provide vector-based semantic search and full-text search (FTS) within indexed knowledge chunks.
• Object Storage (MinIO/S3/OSS supported via OpenDAL): Efficiently handle large binary blobs such as uploaded documents and extracted assets.
• Redis: Acts as a message broker and cache for task coordination and transient data.

The layered design allows modular deployment and scalability. Each storage type is abstracted behind interface classes to allow switching implementations via configuration.

System Hierarchy and Code Entity Integration

Sources: <FileRef file-url="https://github.com/infiniflow/ragflow/blob/53afc323/common/settings.py#L73-L88" min=73 max=88 file-path="common/settings.py">Hii</FileRef>, <FileRef file-url="https://github.com/infiniflow/ragflow/blob/53afc323/common/settings.py#L133-L134" min=133 max=134 file-path="common/settings.py">Hii</FileRef>, <FileRef file-url="https://github.com/infiniflow/ragflow/blob/53afc323/rag/utils/infinity_conn.py#L44-L44" min=44 file-path="rag/utils/infinity_conn.py">Hii</FileRef>, <FileRef file-url="https://github.com/infiniflow/ragflow/blob/53afc323/rag/utils/es_conn.py#L62-L62" min=62 file-path="rag/utils/es_conn.py">Hii</FileRef>, <FileRef file-url="https://github.com/infiniflow/ragflow/blob/53afc323/rag/utils/ob_conn.py#L34-L34" min=34 file-path="rag/utils/ob_conn.py">Hii</FileRef>, <FileRef file-url="https://github.com/infiniflow/ragflow/blob/53afc323/rag/utils/opensearch_conn.py#L64-L65" min=64 max=65 file-path="rag/utils/opensearch_conn.py">Hii</FileRef>, <FileRef file-url="https://github.com/infiniflow/ragflow/blob/53afc323/rag/utils/minio_conn.py#L42-L42" min=42 file-path="rag/utils/minio_conn.py">Hii</FileRef>, <FileRef file-url="https://github.com/infiniflow/ragflow/blob/53afc323/rag/utils/s3_conn.py#L28-L28" min=28 file-path="rag/utils/s3_conn.py">Hii</FileRef>, <FileRef file-url="https://github.com/infiniflow/ragflow/blob/53afc323/rag/utils/redis_conn.py#L61-L61" min=61 file-path="rag/utils/redis_conn.py">Hii</FileRef>

---

Document Store Layer

Overview

The document store forms the core retrieval backend for RAGFlow, supporting hybrid semantic and keyword search over knowledge chunks. It indexes vector embeddings and textual metadata with specialized analyzers for enhanced retrieval quality.

Supported Engines and Key Implementations

Engine	Key Class / File	Special Features
Infinity	InfinityConnection rag/utils/infinity_conn.py44	Table-per-KB separation, custom Rag analyzers, field mapping rag/utils/infinity_conn.py60-75 retry logic for metadata contention common/doc_store/infinity_conn_base.py108-153
Elasticsearch	ESConnection rag/utils/es_conn.py62	Deep pagination with search_after rag/utils/es_conn.py70-134 complex bool queries rag/utils/es_conn.py158-188
OceanBase	OBConnection rag/utils/ob_conn.py34	SQL vector search, specialized column definitions using SQLAlchemy rag/utils/ob_conn.py56-101
OpenSearch	OSConnection rag/utils/opensearch_conn.py64-65	OpenSearch 2.10+ hybrid search pipeline support rag/utils/opensearch_conn.py106-153

The chosen engine is configured via environment variable DOC_ENGINE common/settings.py85 The document store connection singleton interfaces with the appropriate backend transparently.

Document Store Key Features

• Vector-based Search: Engines support vector similarity for hybrid search via expressions like MatchDenseExpr common/doc_store/doc_store_base.py25 ESConnection builds these using Q("knn", ...) or Q("script_score", ...) rag/utils/es_conn.py246-278
• Full-text Search: Specialized text analyzers (e.g., rag-coarse, rag-fine in Infinity) improve precision conf/infinity_mapping.json10-17 InfinityConnection maps these fields during search rag/utils/infinity_conn.py78-104
• Index Separation: Infinity implementation removes kb_id filters for chunk tables as they use table separation per KB rag/utils/infinity_conn.py171-173
• Rank Feature Integration: Fields such as pagerank_fea and tag_feas allow custom ranking signal fusion during retrieval conf/infinity_mapping.json31-32 ESConnection handles atomic pagerank adjustments via scripts rag/utils/es_conn.py36-58

Metadata & Schema

The stored fields and their analyzers are defined in JSON mapping files, e.g., conf/infinity_mapping.json. Important points:

• Fields ending with _kwd (keyword fields) are treated as keyword lists for exact matching rag/utils/infinity_conn.py54-58
• _JSON_LIST_FIELDS in Infinity defines which fields are stored as JSON arrays, such as source_doc_ids and source_chunk_ids rag/utils/infinity_conn.py29-40
• OceanBase uses explicit SQLAlchemy column definitions for chunk tables, including LONGTEXT for content and ARRAY for keywords rag/utils/ob_conn.py56-101

Sources: <FileRef file-url="https://github.com/infiniflow/ragflow/blob/53afc323/conf/infinity_mapping.json#L1-L80" min=1 max=80 file-path="conf/infinity_mapping.json">Hii</FileRef>, <FileRef file-url="https://github.com/infiniflow/ragflow/blob/53afc323/rag/utils/infinity_conn.py#L29-L173" min=29 max=173 file-path="rag/utils/infinity_conn.py">Hii</FileRef>, <FileRef file-url="https://github.com/infiniflow/ragflow/blob/53afc323/rag/utils/ob_conn.py#L56-L101" min=56 max=101 file-path="rag/utils/ob_conn.py">Hii</FileRef>, <FileRef file-url="https://github.com/infiniflow/ragflow/blob/53afc323/rag/utils/es_conn.py#L36-L134" min=36 max=134 file-path="rag/utils/es_conn.py">Hii</FileRef>, <FileRef file-url="https://github.com/infiniflow/ragflow/blob/53afc323/common/doc_store/infinity_conn_base.py#L108-L153" min=108 max=153 file-path="common/doc_store/infinity_conn_base.py">Hii</FileRef>

---

Document Metadata Management

Document metadata, including tenant-specific info and document-level descriptors, are stored in document store indices. The DocMetadataService class manages this, using a tenant-specific index name like ragflow_doc_meta_{tenant_id} api/db/services/doc_metadata_service.py70-80

The meta_filter utility common/metadata_utils.py30-150 provides in-memory filtering for metadata, supporting operators like contains, in, start with, and standard comparisons common/metadata_utils.py96-123 It handles complex logic such as date normalization and strict format detection common/metadata_utils.py54-68

Sources: <FileRef file-url="https://github.com/infiniflow/ragflow/blob/53afc323/api/db/services/doc_metadata_service.py#L70-L80" min=70 max=80 file-path="api/db/services/doc_metadata_service.py">Hii</FileRef>, <FileRef file-url="https://github.com/infiniflow/ragflow/blob/53afc323/common/metadata_utils.py#L30-L150" min=30 max=150 file-path="common/metadata_utils.py">Hii</FileRef>

---

Relational Database Layer: MySQL/PostgreSQL

MySQL or PostgreSQL is used primarily for relational metadata:

• User and Tenant Management: Managed via UserService and TenantService api/apps/services/dataset_api_service.py31-32
• Knowledge Base Metadata: KnowledgebaseService handles CRUD for datasets, including embedding model configuration api/apps/services/dataset_api_service.py90-143
• Document Status: DocumentService tracks document parsing states (UNSTART, RUNNING, DONE, FAIL) api/apps/restful_apis/chunk_api.py126-132

Peewee ORM models define the schema, such as the File model api/apps/services/dataset_api_service.py24 and Task model api/apps/restful_apis/chunk_api.py34

Sources: <FileRef file-url="https://github.com/infiniflow/ragflow/blob/53afc323/api/apps/services/dataset_api_service.py#L90-L143" min=90 max=143 file-path="api/apps/services/dataset_api_service.py">Hii</FileRef>, <FileRef file-url="https://github.com/infiniflow/ragflow/blob/53afc323/api/apps/restful_apis/chunk_api.py#L126-L138" min=126 max=138 file-path="api/apps/restful_apis/chunk_api.py">Hii</FileRef>

---

Object Storage Layer: MinIO / S3 / OSS / OpenDAL

Binary data such as raw document files and extracted assets are stored using an object storage system abstracted by the StorageFactory common/settings.py202-216

Implementations:

• MinIO: Implements RAGFlowMinio, supporting secure connections and bucket prefixing rag/utils/minio_conn.py42-111 The use_default_bucket and use_prefix_path decorators ensure consistent pathing across physical buckets rag/utils/minio_conn.py52-90
• AWS S3: RAGFlowS3 uses boto3 and supports custom endpoint URLs and addressing styles (e.g., path-style vs virtual-hosted) rag/utils/s3_conn.py28-96
• OpenDAL: OpenDALStorage provides a unified interface, including a mysql scheme that stores blobs in a relational table named opendal_storage rag/utils/opendal_conn.py26-39 It manages database configuration like max_allowed_packet to handle large file writes rag/utils/opendal_conn.py96-109

Sources: <FileRef file-url="https://github.com/infiniflow/ragflow/blob/53afc323/rag/utils/minio_conn.py#L42-L111" min=42 max=111 file-path="rag/utils/minio_conn.py">Hii</FileRef>, <FileRef file-url="https://github.com/infiniflow/ragflow/blob/53afc323/rag/utils/s3_conn.py#L28-L96" min=28 max=96 file-path="rag/utils/s3_conn.py">Hii</FileRef>, <FileRef file-url="https://github.com/infiniflow/ragflow/blob/53afc323/rag/utils/opendal_conn.py#L23-L124" min=23 max=124 file-path="rag/utils/opendal_conn.py">Hii</FileRef>

---

Redis Coordination and Caching Layer

Redis (or Valkey) serves as the central cache, distributed lock manager, and task queue broker:

• Task Management: TaskService uses Redis to queue and track background tasks api/apps/restful_apis/chunk_api.py40
• Distributed Locking: Implemented to prevent race conditions during concurrent metadata operations common/doc_store/infinity_conn_base.py36-53
• Health Monitoring: Component health is verified via Redis connectivity rag/utils/redis_conn.py61

Sources: <FileRef file-url="https://github.com/infiniflow/ragflow/blob/53afc323/api/apps/restful_apis/chunk_api.py#L40-L40" min=40 file-path="api/apps/restful_apis/chunk_api.py">Hii</FileRef>, <FileRef file-url="https://github.com/infiniflow/ragflow/blob/53afc323/common/doc_store/infinity_conn_base.py#L36-L53" min=36 max=53 file-path="common/doc_store/infinity_conn_base.py">Hii</FileRef>

---

Data Flow: Natural Language Space to Code Entities

Retrieval and Storage Operation Mapping

Sources: <FileRef file-url="https://github.com/infiniflow/ragflow/blob/53afc323/rag/nlp/search.py#L65-L65" min=65 file-path="rag/nlp/search.py">Hii</FileRef>, <FileRef file-url="https://github.com/infiniflow/ragflow/blob/53afc323/api/apps/restful_apis/chunk_api.py#L181-L200" min=181 max=200 file-path="api/apps/restful_apis/chunk_api.py">Hii</FileRef>, <FileRef file-url="https://github.com/infiniflow/ragflow/blob/53afc323/api/apps/services/dataset_api_service.py#L174-L185" min=174 max=185 file-path="api/apps/services/dataset_api_service.py">Hii</FileRef>

Storage Connectivity and Health Monitoring

Sources: <FileRef file-url="https://github.com/infiniflow/ragflow/blob/53afc323/rag/utils/minio_conn.py#L119-L142" min=119 max=142 file-path="rag/utils/minio_conn.py">Hii</FileRef>, <FileRef file-url="https://github.com/infiniflow/ragflow/blob/53afc323/rag/utils/opendal_conn.py#L76-L78" min=76 max=78 file-path="rag/utils/opendal_conn.py">Hii</FileRef>, <FileRef file-url="https://github.com/infiniflow/ragflow/blob/53afc323/rag/utils/s3_conn.py#L115-L124" min=115 max=124 file-path="rag/utils/s3_conn.py">Hii</FileRef>, <FileRef file-url="https://github.com/infiniflow/ragflow/blob/53afc323/rag/utils/opensearch_conn.py#L161-L164" min=161 max=164 file-path="rag/utils/opensearch_conn.py">Hii</FileRef>

---

Summary

RAGFlow employs a robust polyglot persistence architecture aligning storage technology with workload characteristics:

• Relational DBs for user/tenant metadata and transactional data.
• Document Stores for semantic vector and keyword-based retrieval.
• Object Storage for storing bulky raw document blobs and assets.
• Redis for coordination, caching, and task queuing.

Abstractions such as DocStoreConnection, StorageFactory, and REDIS_CONN encapsulate backend-specific logic, enabling configuration-driven flexibility and extensibility.

Sources: <FileRef file-url="https://github.com/infiniflow/ragflow/blob/53afc323/common/settings.py#L73-L95" min=73 max=95 file-path="common/settings.py">Hii</FileRef>, <FileRef file-url="https://github.com/infiniflow/ragflow/blob/53afc323/common/settings.py#L202-L216" min=202 max=216 file-path="common/settings.py">Hii</FileRef>, <FileRef file-url="https://github.com/infiniflow/ragflow/blob/53afc323/rag/utils/redis_conn.py#L61-L61" min=61 file-path="rag/utils/redis_conn.py">Hii</FileRef>