Hierarchical Agentic Tree Retrieval-Augmented Generation (HAT-RAG)

Our Journey from Vectorless RAG to Building Custom Telecom RAG

In our recent Token Optimization consulting engagement with a telecom client, their team was excited about vectorless RAG implementations. They had already achieved promising results in certain use cases such as Contract Lifecycle Management agents and Technical Documentation authoring agents. Their experience with vectorless RAG which relies on hierarchical document trees, structured navigation and LLM-driven reasoning over natural sections rather than vector embeddings, sparked deeper discussions. We came back and extrapolated these successes to broader enterprise challenges and went deep in.

Enterprise Use Case Where Vectorless RAG Shines

For a telecommunication software vendor dealing with long, highly structured documents such as detailed technical specifications, multi-version API/ architecture documentation or standards-compliant contracts, vectorless RAG excels. It preserved full context by avoiding arbitrary chunking, follows logical hierarchies (e.g., navigating from high-level system architecture to specific module interfaces or clause dependencies), and delivered auditable, deterministic paths. This is especially valuable in contract lifecycle management and technical documentation use cases, where precision and structure directly impact development velocity & compliance.

The client reported strong initial results for precise, section-specific queries within well-formatted technical PDFs and contract repositories. No more semantic drift from embeddings; Retrieval felt logical and far more reliable than traditional approaches for their domain-specific

Uncovering the Limitations

However, as they scaled beyond isolated use cases, limitations emerged quickly. Vectorless approaches thrive on clean, consistent hierarchies but struggle with heterogeneous data sources such as a mix of scanned PDFs, evolving API docs, code snippets, emails and multi-format collections common in any given enterprise environment. Iterative reasoning loops increased latency and compute costs. Performance dropped on noisy data, cross-referential documents spanning multiple files or queries requiring broad coverage and multi-hop synthesis across the corpus. Generalization was poor; small changes in document structure or new release versions broke navigation.

This prompted us at GraaS to experiment internally. We needed something that retained vectorless strengths (structure awareness, reduced hallucination via precise navigation) while adding robustness for real-world enterprise messiness.

Introducing HAT-RAG: Hierarchical Agentic Tree Retrieval-Augmented Generation

We developed HAT-RAG, a hybrid architecture unifying reasoning-driven navigation with hybrid recall and adaptive abstraction. It combines:

– Structure-aware tree indexing (from vectorless) for hierarchical documents.

– Hybrid retrieval (vector + lexical/sparse) for broad, heterogeneous coverage.

– Agentic orchestration with hierarchical decomposition for multi-hop reasoning and adaptive routing.

– Adaptive abstraction layers that summarize or expand context dynamically.

By blending dense retrieval where semantic fuzziness helps, sparse/keyword where precision matters, and agentic tree traversal for deep navigation, HAT-RAG compensates for individual weaknesses. It delivers scalable, robust grounding while maintaining explainability.

Isn’t This Same as Ontology-Grounded Retrieval Augmented Generation and GraphRAG?

One of our client’s AIOps leader pushed back: “Don’t we already have OG-RAG and GraphRAG? Why introduce yet another variant?” This was a fair and important question, which led us to this blog based on month long experiments and real-world evaluations. Let’s compare them objectively and technically.

– OG-RAG (Ontology-Guided RAG): Uses predefined ontologies to constrain and guide queries (e.g., mapping terms to concepts, enforcing relationships via schema). Strengths: High precision on well-modeled domains, excellent explainability, reduced ambiguity through logical inference. Retrieval is structure- and ontology-aware. Weaknesses: High upfront ontology engineering/maintenance cost, limited generalization outside the ontology’s scope and brittleness to ontology mismatches or evolving data

– GraphRAG: Builds a knowledge graph of entities and relationships, often on top of or alongside vector retrieval. Excels at multi-hop reasoning and relational queries by traversing edges. Strengths: Global context synthesis, relationship awareness. Weaknesses: High upfront graph construction cost, maintenance overhead.

– HAT-RAG (Our Hybrid): Starts with hierarchical tree structures for documents (vectorless-inspired), augments with hybrid recall (vector + sparse), and layers agentic control for decomposition, routing, verification, and adaptive abstraction.

Critical differences:

1. Core Primitive: Dynamic tree navigation + hybrid recall + agentic hierarchy, versus ontology/schema-driven guidance (OG-RAG) or entity graphs (GraphRAG).
2. Flexibility vs. Rigidity: HAT-RAG’s LLM-driven tree construction and agentic reasoning adapt to varied or imperfect structures without requiring a comprehensive upfront ontology. OG-RAG shines when a quality of ontology is high and it is evolving at least near-time.
3. Handling Hierarchies & Multi-Hop: Natively preserves document trees while using agents for dynamic multi-hop; complements but does not strictly depend on formal ontologies.
4. Trade-offs Addressed: Mitigates OG-RAG’s engineering overhead and GraphRAG’s build cost by being selective such as using trees + agents where structure exists and hybrid fallbacks elsewhere. Better generalization across mixed data.

In HAT-RAG, reasoning happens live, at query time, every single query. The tree/hybrid-recall gives it candidates, but an LLM has to iteratively decide “is this enough, or do I retrieve more”. Nothing is precomputed to answer what to retrieve; only the index structure (the tree) is precomputed.

Use Case Fitment

–  OG-RAG wins where: an ontology already exists or is cheap to build, query volume is high, traceability is legally and operationally mandatory, and the domain doesn’t drift fast enough to make the ontology stale. Healthcare, legal, drug-discovery, telecommunication and industrial utility scale

-HAT-RAG is perfect fit where corpus is too heterogeneous, fast-moving or query volume is comparatively low but the value of any single correct answer is very high, and multi-hop synthesis across mismatched document types is the actual job. M&A / PE due diligence, MNOs, legal e-discovery/litigation research, scientific literature review / R&D competitive intelligence and enterprise functions such as HR and Procurement.

Performance Evaluation of HAT-RAG

We also evaluated performance scaling with corpus volume for 3 RAG types keeping other components in the AI-value chain constant. Here are the results:
                                                         

Our internal experiments and 2 client pilots confirm that HAT-RAG definitely advances RAG reliability. HAT-RAG represents a practical step forward in making RAG systems more reliable and scalable for complex, real-world enterprise environments. It certainly delivers better precision, adaptability and explainability where vectorless approaches fall short.

GraaS is never not about inventing novelty for novelty’s sake but solving real production gaps and business value. If your are grappling with similar challenges, we’d love to hear your experiences or explore how these patterns might apply to your challenges, feel free to share your thoughts or reach out to us at [email protected]. Probably we can solve it together.