This document describes RAGFlow's document store abstraction layer, which provides a unified interface for interacting with multiple search engine backends. The abstraction enables RAGFlow to support Elasticsearch, Infinity, OceanBase, and OpenSearch as pluggable storage engines without modifying application code. This layer handles chunk storage, vector indexing, full-text search, and hybrid retrieval operations.
The abstraction ensures that the RAG pipeline remains agnostic of the underlying storage technology, whether it is a traditional search engine (Elasticsearch), a high-performance AI-native database (Infinity), or a distributed relational database with vector capabilities (OceanBase).
The document store abstraction follows a three-tier architecture: an abstract interface, backend-specific base classes, and concrete implementations selected at runtime via environment configuration. In addition to the Python implementation, a Go-based DocEngine interface provides similar functionality for the Go server components.
Diagram: Document Store Abstraction Layers
Sources: <FileRef file-url="https://github.com/infiniflow/ragflow/blob/53afc323/common/doc_store/doc_store_base.py#L34-L34" min=34 file-path="common/doc_store/doc_store_base.py">Hii</FileRef>, <FileRef file-url="https://github.com/infiniflow/ragflow/blob/53afc323/common/settings.py#L85-L91" min=85 max=91 file-path="common/settings.py">Hii</FileRef>, <FileRef file-url="https://github.com/infiniflow/ragflow/blob/53afc323/rag/utils/infinity_conn.py#L44-L44" min=44 file-path="rag/utils/infinity_conn.py">Hii</FileRef>, <FileRef file-url="https://github.com/infiniflow/ragflow/blob/53afc323/rag/utils/es_conn.py#L62-L62" min=62 file-path="rag/utils/es_conn.py">Hii</FileRef>, <FileRef file-url="https://github.com/infiniflow/ragflow/blob/53afc323/rag/utils/ob_conn.py#L34-L42" min=34 max=42 file-path="rag/utils/ob_conn.py">Hii</FileRef>
The DocStoreConnection abstract base class defines the contract that all backend implementations must satisfy. It is organized into functional categories for index management, CRUD, and retrieval.
| Method | Parameters | Purpose |
|---|---|---|
db_type() | None | Returns backend type identifier ("elasticsearch", "infinity", "oceanbase", "opensearch") |
health() | None | Returns cluster health status including availability and resource metrics |
create_idx() | indexName, knowledgebaseId, vectorSize, parser_id | Creates index/table with schema for given vector dimensions |
delete_idx() | indexName, knowledgebaseId | Drops index/table |
index_exist() | indexName, knowledgebaseId | Checks if index/table exists |
create_doc_meta_idx() | index_name | Creates a dedicated index for document metadata |
refresh_idx() | index_name | Refreshes an index to make recent changes visible |
Sources: <FileRef file-url="https://github.com/infiniflow/ragflow/blob/53afc323/common/doc_store/doc_store_base.py#L143-L185" min=143 max=185 file-path="common/doc_store/doc_store_base.py">Hii</FileRef>, <FileRef file-url="https://github.com/infiniflow/ragflow/blob/53afc323/rag/utils/infinity_conn.py#L132-L136" min=132 max=136 file-path="rag/utils/infinity_conn.py">Hii</FileRef>
The search() method is the primary entry point for retrieval, supporting hybrid search, filtering, and aggregation.
Key Parameters:
condition: A dictionary of filter criteria (e.g., {"available_int": 1}).match_expressions: A list of MatchTextExpr (BM25), MatchDenseExpr (Vector), or FusionExpr to execute.knowledgebase_ids: Used for multi-tenancy partitioning or table selection.Sources: <FileRef file-url="https://github.com/infiniflow/ragflow/blob/53afc323/common/doc_store/doc_store_base.py#L191-L236" min=191 max=236 file-path="common/doc_store/doc_store_base.py">Hii</FileRef>, <FileRef file-url="https://github.com/infiniflow/ragflow/blob/53afc323/rag/utils/infinity_conn.py#L110-L123" min=110 max=123 file-path="rag/utils/infinity_conn.py">Hii</FileRef>, <FileRef file-url="https://github.com/infiniflow/ragflow/blob/53afc323/rag/utils/es_conn.py#L136-L149" min=136 max=149 file-path="rag/utils/es_conn.py">Hii</FileRef>
Infinity uses a table-per-knowledgebase strategy for chunk storage. It performs significant field mapping to align RAGFlow's logical fields with physical database columns.
Field Conversion Logic:
Infinity maps logical RAGFlow fields to specific columns and full-text indexes defined in conf/infinity_mapping.json.
| RAGFlow Field | Infinity Column | Full-Text Index |
|---|---|---|
docnm_kwd | docnm | ft_docnm_rag_coarse |
content_ltks | content | ft_content_rag_coarse |
important_kwd | important_keywords | ft_important_keywords_rag_coarse |
Implementation Details:
kb_id filters are removed from conditions because the storage is separated by table <FileRef file-url="https://github.com/infiniflow/ragflow/blob/53afc323/rag/utils/infinity_conn.py#L172-L173" min=172 max=173 file-path="rag/utils/infinity_conn.py">Hii</FileRef>._retry_on_meta_contention with exponential backoff + jitter to handle RocksDB transaction contention (error 9003) during table/index creation <FileRef file-url="https://github.com/infiniflow/ragflow/blob/53afc323/common/doc_store/infinity_conn_base.py#L108-L145" min=108 max=145 file-path="common/doc_store/infinity_conn_base.py">Hii</FileRef>.kb_id and available_int <FileRef file-url="https://github.com/infiniflow/ragflow/blob/53afc323/conf/infinity_mapping.json#L4-L28" min=4 max=28 file-path="conf/infinity_mapping.json">Hii</FileRef>.source_doc_ids and source_chunk_ids are stored as JSON lists to support reference-counting during deletion <FileRef file-url="https://github.com/infiniflow/ragflow/blob/53afc323/rag/utils/infinity_conn.py#L29-L40" min=29 max=40 file-path="rag/utils/infinity_conn.py">Hii</FileRef>, <FileRef file-url="https://github.com/infiniflow/ragflow/blob/53afc323/conf/infinity_mapping.json#L48-L49" min=48 max=49 file-path="conf/infinity_mapping.json">Hii</FileRef>.Sources: <FileRef file-url="https://github.com/infiniflow/ragflow/blob/53afc323/rag/utils/infinity_conn.py#L60-L105" min=60 max=105 file-path="rag/utils/infinity_conn.py">Hii</FileRef>, <FileRef file-url="https://github.com/infiniflow/ragflow/blob/53afc323/conf/infinity_mapping.json#L9-L17" min=9 max=17 file-path="conf/infinity_mapping.json">Hii</FileRef>, <FileRef file-url="https://github.com/infiniflow/ragflow/blob/53afc323/common/doc_store/infinity_conn_base.py#L108-L145" min=108 max=145 file-path="common/doc_store/infinity_conn_base.py">Hii</FileRef>
Elasticsearch stores chunks in a single tenant-wide index, using the kb_id field for logical partitioning.
Key Features:
<FileRef file-url="https://github.com/infiniflow/ragflow/blob/53afc323/rag/utils/es_conn.py#L183-L205" min=183 max=205 file-path="rag/utils/es_conn.py">Hii</FileRef>._search_with_search_after() to efficiently page through large result sets beyond the 10,000 document MAX_RESULT_WINDOW <FileRef file-url="https://github.com/infiniflow/ragflow/blob/53afc323/rag/utils/es_conn.py#L70-L134" min=70 max=134 file-path="rag/utils/es_conn.py">Hii</FileRef>._PAGERANK_FEA_ADJUST_SCRIPT) for atomic updates to chunk relevance scores, clamping values between min_w and max_w <FileRef file-url="https://github.com/infiniflow/ragflow/blob/53afc323/rag/utils/es_conn.py#L36-L58" min=36 max=58 file-path="rag/utils/es_conn.py">Hii</FileRef>.Sources: <FileRef file-url="https://github.com/infiniflow/ragflow/blob/53afc323/rag/utils/es_conn.py#L31-L134" min=31 max=134 file-path="rag/utils/es_conn.py">Hii</FileRef>, <FileRef file-url="https://github.com/infiniflow/ragflow/blob/53afc323/rag/utils/es_conn.py#L183-L205" min=183 max=205 file-path="rag/utils/es_conn.py">Hii</FileRef>
The OceanBase implementation leverages pyobvector and SQLAlchemy for vector and relational data management.
Schema Definition: OceanBase defines a SQL schema including:
ARRAY types <FileRef file-url="https://github.com/infiniflow/ragflow/blob/53afc323/rag/utils/ob_conn.py#L68-L72" min=68 max=72 file-path="rag/utils/ob_conn.py">Hii</FileRef>.pagerank_fea (Integer) and tag_feas (JSON) <FileRef file-url="https://github.com/infiniflow/ragflow/blob/53afc323/rag/utils/ob_conn.py#L67-L73" min=67 max=73 file-path="rag/utils/ob_conn.py">Hii</FileRef>.n_hop_with_weight <FileRef file-url="https://github.com/infiniflow/ragflow/blob/53afc323/rag/utils/ob_conn.py#L54-L55" min=54 max=55 file-path="rag/utils/ob_conn.py">Hii</FileRef>.available_int with a server default of "1" <FileRef file-url="https://github.com/infiniflow/ragflow/blob/53afc323/rag/utils/ob_conn.py#L74-L74" min=74 file-path="rag/utils/ob_conn.py">Hii</FileRef>.FTS_COLUMNS_TKS including title_tks and content_ltks <FileRef file-url="https://github.com/infiniflow/ragflow/blob/53afc323/rag/utils/ob_conn.py#L126-L133" min=126 max=133 file-path="rag/utils/ob_conn.py">Hii</FileRef>.Sources: <FileRef file-url="https://github.com/infiniflow/ragflow/blob/53afc323/rag/utils/ob_conn.py#L56-L101" min=56 max=101 file-path="rag/utils/ob_conn.py">Hii</FileRef>, <FileRef file-url="https://github.com/infiniflow/ragflow/blob/53afc323/rag/utils/ob_conn.py#L118-L133" min=118 max=133 file-path="rag/utils/ob_conn.py">Hii</FileRef>
RAGFlow uses a uniform expression language to specify search queries, enabling backend-agnostic processing.
Represents textual keyword search specifying fields and matching text <FileRef file-url="https://github.com/infiniflow/ragflow/blob/53afc323/common/doc_store/doc_store_base.py#L56-L68" min=56 max=68 file-path="common/doc_store/doc_store_base.py">Hii</FileRef>.
Represents vector similarity searches targeting numeric embedding columns <FileRef file-url="https://github.com/infiniflow/ragflow/blob/53afc323/common/doc_store/doc_store_base.py#L70-L86" min=70 max=86 file-path="common/doc_store/doc_store_base.py">Hii</FileRef>.
Enables composite searches combining multiple expressions weighted for hybrid retrieval <FileRef file-url="https://github.com/infiniflow/ragflow/blob/53afc323/common/doc_store/doc_store_base.py#L114-L127" min=114 max=127 file-path="common/doc_store/doc_store_base.py">Hii</FileRef>.
Sources: <FileRef file-url="https://github.com/infiniflow/ragflow/blob/53afc323/common/doc_store/doc_store_base.py#L56-L127" min=56 max=127 file-path="common/doc_store/doc_store_base.py">Hii</FileRef>
DocMetadataService manages document-level metadata (e.g., titles, authors) stored in the document store, serving as the sole source of truth for RAG-relevant metadata.
ragflow_doc_meta_{tenant_id} for metadata storage.meta_filter from common/metadata_utils.py to apply complex comparison operators like contains, start with, and date comparisons. It supports case-insensitive matching and handles various value types <FileRef file-url="https://github.com/infiniflow/ragflow/blob/53afc323/common/metadata_utils.py#L30-L150" min=30 max=150 file-path="common/metadata_utils.py">Hii</FileRef>.convert_conditions transforms application-level filters into the format expected by the DocStoreConnection.search method <FileRef file-url="https://github.com/infiniflow/ragflow/blob/53afc323/common/metadata_utils.py#L153-L178" min=153 max=178 file-path="common/metadata_utils.py">Hii</FileRef>.Sources: <FileRef file-url="https://github.com/infiniflow/ragflow/blob/53afc323/common/metadata_utils.py#L30-L178" min=30 max=178 file-path="common/metadata_utils.py">Hii</FileRef>, <FileRef file-url="https://github.com/infiniflow/ragflow/blob/53afc323/api/apps/restful_apis/chunk_api.py#L35-L36" min=35 max=36 file-path="api/apps/restful_apis/chunk_api.py">Hii</FileRef>
The system initializes the document store connection at startup based on the DOC_ENGINE environment variable.
Diagram: System Initialization Flow
Sources: <FileRef file-url="https://github.com/infiniflow/ragflow/blob/53afc323/common/settings.py#L85-L91" min=85 max=91 file-path="common/settings.py">Hii</FileRef>, <FileRef file-url="https://github.com/infiniflow/ragflow/blob/53afc323/common/settings.py#L29-L32" min=29 max=32 file-path="common/settings.py">Hii</FileRef>, <FileRef file-url="https://github.com/infiniflow/ragflow/blob/53afc323/common/settings.py#L218-L245" min=218 max=245 file-path="common/settings.py">Hii</FileRef>
Refresh this wiki
This wiki was recently refreshed. Please wait 4 days to refresh again.