The Work

Baibelfish is the document ingestion engine - the system that transforms raw content into a form the platform can retrieve from. It processes 12 input formats (PDF, HTML, DOCX, CSV, and others) using format-specific extraction pipelines. The critical design decision is content-aware chunking: rather than splitting documents by size or delimiter, Baibelfish analyses document structure, semantic density, and heading hierarchy to produce chunks that are optimised for retrieval accuracy rather than processing convenience.

After chunking, the engine runs entity extraction, cross-document relationship mapping, and local embedding generation - building both a vector index and a Dgraph-based knowledge graph. RAPTOR hierarchical summarisation creates multi-level abstractions that support both precise retrieval and broad thematic queries.

DeepThought introduces the staging-and-promotion paradigm to RAG ingestion. Rather than processing documents in a single undifferentiated pass, it moves content through defined stages - raw, parsed, chunked, enriched, validated, expert-promoted - with transactional transitions between each. Stage transitions are atomic: a document that fails validation stays at its current stage and does not corrupt the retrieval index. Rollback is possible. Audit trails are maintained.

Expert promotion identifies documents of particularly high retrieval value - primary sources, authoritative references, high-confidence extractions - and gives them elevated weight in the retrieval layer. The evaluation demonstrates measurable retrieval quality uplift from promotion, quantified across 35,000+ document sections.

Every incoming query is classified before retrieval begins. The classifier - an ONNX-compiled neural network running on local hardware - assigns one of seven query types: factual, analytical, comparative, temporal, multi-hop, ambiguous, or out-of-scope. The classification governs downstream routing: which retrieval strategy is used, whether the knowledge graph is queried alongside the vector index, whether RAPTOR summaries are relevant, and what citation standard applies.

Classification takes 10-30ms and produces no API traffic. Out-of-scope queries are handled at this stage - returning a boundary response without an LLM call - which alone accounts for a significant fraction of the token savings on real-world workloads.

Retrieval combines dense vector search with traversal of a Dgraph-based knowledge graph. The vector index returns semantically similar chunks; the knowledge graph adds entity relationships, cross-document connections, and provenance chains that vector similarity alone cannot surface. For multi-hop queries - where the answer requires connecting information across multiple documents - graph traversal is essential.

Retrieved candidates are reranked using a local neural network before context assembly. The reranking model is domain-tuned per tenant where training data is available.

Before any content reaches the LLM, it passes through a semantic compression stage. Retrieved chunks are compressed - removing redundancy, low-information passages, and content already represented in the existing context - at ratios between 16× and 64× depending on document type. Only the minimum context required to answer the query is assembled into the prompt.

The citation system assigns one of three citation types to each piece of retrieved content: direct quote, paraphrase with confidence score, or inferred relationship. These citations survive into the generated response - the LLM is given the citation structure, not asked to construct it from memory.

This stage is where the majority of the token savings materialise. On a standard RAG pipeline, the full retrieved context would go to the LLM unchanged. Here, only what is needed goes - and the structure of what is needed is explicitly defined.

The conversation context system manages memory across sessions using Ebbinghaus forgetting curves - a mathematical model of how human memory decays over time. Information from earlier in a conversation fades at a calibrated rate; corrections and high-confidence new information are retained at higher weight. The system does not have a fixed context window; it has a decaying memory with active management.

AI character state is modelled using the PAD emotional model (Pleasure, Arousal, Dominance) - a well-validated psychological framework for representing emotional state as a three-dimensional space. Emotional state shifts in response to conversational events and influences tone and response strategy. No biometrics are used; all emotional inference is from text.

Communication Accommodation Theory governs how the system adapts to individual users over time - adjusting register, vocabulary, and pace. This is believed to be the first production deployment of Communication Accommodation Theory in an AI system.