This document explains the technical implementation and use cases for the various chunking strategies in RAGFlow. Chunking is the process of splitting parsed document content into discrete, semantically meaningful segments for indexing and retrieval. RAGFlow provides a variety of specialized chunking modes, each optimized for specific document structures like legal articles, academic papers, or spreadsheet rows.
For information about document parsing before chunking, see 6.1 Document Parsing Strategies For information about embedding generation after chunking, see 6.3 Content Enhancement and Embedding
RAGFlow implements chunking strategies as specialized modules within the rag/app/ directory. Each module contains a chunk() function that orchestrates the parsing and splitting logic.
| Mode | Module | Primary Use Case | Key Features |
|---|---|---|---|
| Naive | naive.py | General documents | Token-aware splitting, delimiter-based, supports most formats rag/app/naive.py763-1028 |
| Manual | manual.py | Structured documents | Preserved hierarchical structure via section levels rag/app/manual.py137-236 |
| Q&A | qa.py | Question-answer pairs | Extracts Q&A pairs using regex patterns rag/app/qa.py319-466 |
| Table | table.py | Structured data | Row-based chunking for Excel/CSV with image support rag/app/table.py42-130 |
| Book | book.py | Long-form content | Hierarchical merging, TOC removal rag/app/book.py63-182 |
| Laws | laws.py | Legal documents | Tree-based structure preserving articles/sections rag/app/laws.py120-217 |
| Paper | paper.py | Academic papers | Metadata extraction (abstract, authors) rag/app/paper.py149-260 |
| Presentation | presentation.py | Slides/presentations | One page per chunk with visual context rag/app/presentation.py128-243 |
| One | one.py | Complete document | Entire file as a single semantic unit rag/app/one.py59-167 |
| Knowledge Graph | knowledge_graph.py | Entity-relationship extraction | Extracts entities and relationships for graph-based RAG. |
Sources: rag/app/naive.py763-1028 rag/app/manual.py137-236 rag/app/qa.py319-466 rag/app/table.py42-130 rag/app/book.py63-182 rag/app/laws.py120-217 rag/app/paper.py149-260 rag/app/presentation.py128-243 rag/app/one.py59-167
The chunking process is orchestrated by the background task executor, which selects the application module based on the parser_id defined in the dataset configuration.
The diagram below shows how the high-level system names (like "Naive Chunking") map to specific code entities and functions within the Python backend.
Diagram: Chunking System to Code Mapping
Sources: rag/app/naive.py115-184 rag/app/manual.py137-173 rag/nlp/__init__.py31 common/token_utils.py21
Naive chunking provides general-purpose document splitting. It supports PDF, DOCX, TXT, Markdown, and HTML.
The parser_config controls the splitting behavior:
| Parameter | Default | Description |
|---|---|---|
chunk_token_num | 512 | Maximum tokens per chunk rag/app/naive.py774 |
delimiter | "\n!?。；！？" | Delimiters for splitting rag/app/naive.py774 |
layout_recognize | "DeepDOC" | Layout analysis engine rag/app/naive.py149 |
Sources: rag/app/naive.py774-786
The naive_merge() function implements splitting that respects token limits while preserving semantic boundaries:
num_tokens_from_string() common/token_utils.py21chunk_token_num.@@page\tleft\tright\ttop\tbottom## tags rag/app/naive.py821-825Sources: rag/nlp/__init__.py1145-1300 common/token_utils.py21
Manual chunking preserves document hierarchy by detecting titles and heading levels.
Manual mode utilizes bullets_category() to detect heading patterns (e.g., Chapter I, 1.1).
PdfParser (as Pdf class) to extract layout blocks and merges them based on vertical position rag/app/manual.py33-67docx_question_level() to build a hierarchical stack of questions and answers based on paragraph styles rag/app/manual.py70-134Sources: rag/app/manual.py33-134 rag/nlp/__init__.py169-233
Q&A chunking extracts question-answer pairs. For PDF files, it uses QUESTION_PATTERN to detect question bullets:
It identifies questions using has_qbullet() which checks for matches against these patterns and ensures sequence continuity rag/nlp/__init__.py179-219
Sources: rag/nlp/__init__.py164-219 rag/app/qa.py98-101
Table chunking treats rows in structured datasets (Excel, CSV) as chunks.
The Excel class in rag/app/table.py handles spreadsheet parsing:
_extract_images_from_worksheet() in RAGFlowExcelParser pulls images from cells deepdoc/parser/excel_parser.py110-153vision_figure_parser_figure_xlsx_wrapper() for LLM-based description rag/app/table.py83-86Sources: rag/app/table.py42-130 deepdoc/parser/excel_parser.py110-153
Presentation mode treats each slide as a single chunk with a thumbnail.
For PDF presentations, the Pdf class in rag/app/presentation.py:
page_number rag/app/presentation.py59-99(top, x0) to maintain visual order rag/app/presentation.py106-107Sources: rag/app/presentation.py35-115
The Laws method is designed for legal documents with rigid structures (Chapters, Sections, Articles).
docx_question_level() identifies the depth of paragraphs rag/app/laws.py109Node class in rag/nlp/__init__.py to build a tree based on detected levels rag/app/laws.py129-130Sources: rag/app/laws.py83-132 rag/nlp/__init__.py28
The Paper chunking method is specifically tailored for academic papers, focusing on extracting key components like title, authors, abstract, and structured sections.
The Pdf class in rag/app/paper.py implements the parsing logic:
Sources: rag/app/paper.py31-146 rag/app/paper.py149-260
The "One" chunking method treats the entire document as a single, contiguous chunk. This is useful for very short documents or when the entire document's context is always required.
The chunk() function in rag/app/one.py aggregates all parsed content:
Sources: rag/app/one.py59-167
attach_media_context() enriches table and image chunks with surrounding text within the configured token budget (table_context_size or image_context_size).
Sources: rag/nlp/__init__.py409-704
VisionFigureParser and its wrappers (e.g., vision_figure_parser_pdf_wrapper) utilize LLMs to generate descriptions for images and tables. In by_mineru, if a tenant has a VISION model configured, it is used to enrich image chunks with VLM-generated semantic descriptions rag/app/naive.py162-167
Diagram: Vision Enhancement Data Flow
Sources: rag/app/naive.py115-128 rag/app/naive.py151-167 rag/app/picture.py91-101
All methods use rag_tokenizer for unified segmentation and token counting.
Diagram: Tokenization Components
Sources: rag/nlp/__init__.py31 rag/nlp/__init__.py302-327 common/token_utils.py21
Refresh this wiki
This wiki was recently refreshed. Please wait 4 days to refresh again.