ArborParser
ArborParser is a powerful Python library designed to parse structured text documents and convert them into a tree representation based on hierarchical headings. It intelligently handles various numbering schemes and document inconsistencies, making it ideal for processing outlines, reports, technical documentation, legal texts, and more.
Features
- Chain Parsing: Converts text into a linear sequence (
ChainNode
list) representing the document's hierarchical structure.
- Flexible Pattern Definition: Define custom parsing patterns using regular expressions and specific number converters (Arabic, Roman, Chinese, Letters, Circled).
- Built-in Patterns: Provides ready-to-use patterns for common heading styles (
1.2.3
, Chapter 1
, 第一章
, etc.).
- Robust Tree Building: Transforms the linear chain into a true hierarchical
TreeNode
structure.
- Automatic Error Correction: Includes an
AutoPruneStrategy
to intelligently handle skipped heading levels or lines mistakenly identified as headings.
- Node Manipulation: Allows merging content between nodes (
concat_node
) for flexibility in handling non-heading text or correcting parsing errors.
- Reversible Transformation: Preserves original text, enabling full document reconstruction from the tree (
tree.get_full_content()
).
- Export Capabilities: Outputs the parsed structure in various formats (e.g., human-readable tree view).
Example Transformation:
Original Text
Chapter 1 Animals
1.1 Mammals
1.1.1 Primates
1.2 Reptiles
Chapter 2 Plants
2.1 Angiosperms
Chain Structure (Intermediate)
LEVEL-[]: ROOT
LEVEL-[1]: Animals
LEVEL-[1, 1]: Mammals
LEVEL-[1, 1, 1]: Primates
LEVEL-[1, 2]: Reptiles
LEVEL-[2]: Plants
LEVEL-[2, 1]: Angiosperms
Tree Structure (Final)
ROOT
├─ Chapter 1 Animals
│ ├─ 1.1 Mammals
│ │ └─ 1.1.1 Primates
│ └─ 1.2 Reptiles
└─ Chapter 2 Plants
└─ 2.1 Angiosperms
Installation
pip install arborparser
Basic Usage
from arborparser.chain import ChainParser
from arborparser.tree import TreeBuilder, TreeExporter, AutoPruneStrategy
from arborparser.pattern import ENGLISH_CHAPTER_PATTERN_BUILDER, NUMERIC_DOT_PATTERN_BUILDER
test_text = """
Chapter 1 Animals
1.1 Mammals
1.1.1 Primates
1.2 Reptiles
Chapter 2 Plants
2.1 Angiosperms
"""
patterns = [
ENGLISH_CHAPTER_PATTERN_BUILDER.build(),
NUMERIC_DOT_PATTERN_BUILDER.build(),
]
parser = ChainParser(patterns)
chain = parser.parse_to_chain(test_text)
builder = TreeBuilder(strategy=AutoPruneStrategy())
tree = builder.build_tree(chain)
print(TreeExporter.export_tree(tree))
Key Features in Detail
Built-in & Custom Patterns
Quickly parse common formats using builders like NUMERIC_DOT_PATTERN_BUILDER
, CHINESE_CHAPTER_PATTERN_BUILDER
, etc., or define your own using PatternBuilder
for full control over prefixes, suffixes, number types, and separators.
letter_section_pattern = PatternBuilder(
prefix_regex=r"Section\s",
number_type=NumberType.LETTER,
suffix_regex=r"\."
).build()
Automatic Error Correction (AutoPruneStrategy)
Documents aren't always perfect. AutoPruneStrategy
(the default for TreeBuilder
) handles common issues like skipped heading numbers (e.g., 1.1
followed by 1.3
) and prunes lines incorrectly matched as headings, ensuring a more robust parsing process compared to the StrictStrategy
.
Okay, here is a dedicated section explaining AutoPruneStrategy
using the provided example, formatted for a README without using Python code blocks for the illustration:
Automatic Error Correction (AutoPruneStrategy)
Real-world documents often contain structural inconsistencies that can challenge parsers. Common issues include:
- Skipped Heading Levels: Authors might jump from
1.1
directly to 1.3
, omitting 1.2
.
- False Positives: Regular text lines might accidentally match a heading pattern (e.g., a sentence mentioning "section 1.1").
The AutoPruneStrategy
(used by default in TreeBuilder
) is designed to handle these imperfections gracefully. It uses heuristics to identify likely errors and prune the intermediate structure, resulting in a more accurate final tree.
Example: Handling Imperfections
Consider the following text with a missing section (1.2
) and a line of text containing 1.1
which could be mistaken for a heading:
Input Text:
Chapter 1 The Foundation
Introductory content for the first chapter.
1.1 Core Concepts
Explanation of the fundamental ideas.
This section lays the groundwork.
# NOTE: Heading '1.2 Intermediate Concepts' is MISSING here.
1.3 Advanced Topics
Discussing more complex subjects. We build upon the ideas from section
1.1. This section is more advanced and goes into more detail.
# NOTE: The '1.1.' here is text, not a heading.
Chapter 2 Building Blocks
Content for the second chapter.
2.1 Component A
Details about the first component.
2.2 Component B
Details about the second component. End of document.
Intermediate Chain (Before Pruning):
A naive parsing step might initially produce a chain like this, including the misidentified heading:
LEVEL-[]: ROOT
LEVEL-[1]: The Foundation
LEVEL-[1, 1]: Core Concepts
LEVEL-[1, 3]: Advanced Topics
LEVEL-[1, 1]: This section is more advanced and goes into more detail. <-- POTENTIAL FALSE POSITIVE
LEVEL-[2]: Building Blocks
LEVEL-[2, 1]: Component A
LEVEL-[2, 2]: Component B
How AutoPrune Works:
When building the tree, AutoPruneStrategy
analyzes the sequence:
- It recognizes that
LEVEL-[1, 3]
can logically follow LEVEL-[1, 1]
even if [1, 2]
is missing (sibling jump).
- It sees the subsequent
LEVEL-[1, 1]
node ("This section...") followed by a completely different hierarchy (LEVEL-[2]
). This discontinuity strongly suggests the second LEVEL-[1, 1]
node was a false positive.
- The strategy "prunes" the misidentified node, effectively merging its content back into the preceding valid node (
LEVEL-[1, 3]
in this case, depending on implementation details of content association).
Final Tree Structure (After AutoPrune):
The resulting tree correctly reflects the intended document structure:
ROOT
├─ Chapter 1 The Foundation
│ ├─ 1.1 Core Concepts
│ └─ 1.3 Advanced Topics # Correctly handles the jump & ignored false positive
└─ Chapter 2 Building Blocks
├─ 2.1 Component A
└─ 2.2 Component B
Node Operations & Reversibility
ArborParser works with ChainNode
(linear sequence) and TreeNode
(hierarchical tree) objects. Both inherit from BaseNode
, which stores level_seq
, title
, and the original content
string.
-
Concatenating Content: You can merge the content of one node into another. This is useful internally for associating non-heading text with its preceding heading or for merging nodes during error correction.
node_a.concat_node(node_b)
-
Merging Children: A parent node can absorb the content of all its descendants.
node_a.merge_all_children()
-
Reconstructing Original Text: Because each node retains its original text chunk (content
), you can reconstruct the entire original document from the root TreeNode
. This verifies parsing integrity and allows regeneration after modification.
reconstructed_text = root_node.get_full_content()
assert reconstructed_text == original_text
Potential Use Cases
- Documentation Parsing
- Legal Document Analysis (Laws, Contracts)
- Outline Processing & Conversion
- Report Structuring & Analysis
- Content Management System Import
- Data Extraction from Structured Text
- Format Conversion (e.g., Text to HTML/XML preserving structure)
- Better Chunking Strategies for RAG
Contributing
Contributions (pull requests, issues) are welcome!
License
MIT License.