AI BriefingTwo Rounds of Trial and Error to Teach AI a Domain — Combining LLM Wiki and RAG
Key point
Shares the process of building an AI system that efficiently delivers domain knowledge by combining LLM Wiki and RAG.
Details
To create a 'senior AI' that understands domain knowledge well, an attempt was made to combine the LLM Wiki pattern with RAG.
The first stage, the Inverted Index approach, used keyword-based indexing, but poor synonym handling and references to irrelevant documents resulted in low token efficiency and accuracy.
The second stage, the body embedding approach, enabled semantic search, but ran into the following limitations.
- Semantic averaging: Vectors of multi-topic documents became ambiguous, degrading search quality
- Weak abbreviation search: Poor recognition of in-house abbreviations or identifiers (IDs)
- Truncation: Information loss due to the embedding model's input limits
Ultimately, this was improved into a structure that divides the work between summary embedding and body FTS (Full-Text Search).
- Semantic matching: Embedding short summaries generated by an LLM to improve search accuracy
- Keyword matching: Using SQLite FTS5 to accurately search abbreviations and identifiers
This process was not simply relying on the model, but a process of reorganizing knowledge into a form that AI can understand well and dividing roles between people and tools.
This summary was generated automatically by AI. Check the original for the author's claims and context. Copyright belongs to the original author.
Our guide explains how the AI works. Report summary errors, attribution issues, or removal requests via Contact.