TL;DR - I run a personal RAG knowledge base over years of my own notes, conversations and docs. Its retrieval was already near the practical ceiling (top-3 relevance around 62%, with a hard ceiling near 64-65%), yet the answers it produced still missed facts and occasionally made things up. The root cause was not retrieval: it was that raw transcripts get truncated when the context is delivered, cutting off mid-fact and inviting the model to guess. The fix was an ETL step, not a ranker change: distill every source into atomic, dated, self-contained claims, index those, keep the raw in cold storage. On the personal corpus, answer usefulness went from 88.6% to 93.9% and groundedness from 84.1% to 93.9%, both statistically significant (McNemar exact, p=0.016 and p=0.024). On a second, larger private corpus a paired A/B put usefulness at +18.8 points and groundedness at +21.3 points (p=0.003 and p<0.001). The corpus shrank about 12x. I then made it permanent with distill-on-ingest so the property maintains itself. Here is the whole thing, including the parts that did not work and the honest caveats.
The job to be done
I keep a single searchable memory over everything I have written and every working session I have had: notes, design docs, chat transcripts, meeting summaries. A retrieval-augmented setup answers questions against it. The recurring failure was not that it retrieved the wrong document. It was that even with the right document in hand, the answer was thin or, worse, confidently wrong.
When I ask my knowledge base a real question, I want an answer that is both useful and actually supported by what I stored, so that I can trust it without re-reading the source every time.
I had been treating this as a retrieval problem and had run out of retrieval road. So I measured the thing I actually cared about instead.
Retrieval was not the bottleneck
The standard retrieval metric (is a relevant document in the top-3?) sat around 62%, and analysis said the real ceiling was 64-65%. Chasing that last couple of points is a saturated investment.
But retrieval quality and answer quality are different things. So I built an answer-level evaluation: for 132 real questions I had actually asked, judge two things about the answer the system would produce - is it useful (does the context let you answer correctly) and is it grounded (would the answer be supported, not guessed). The gap was stark:
| Metric | Score |
|---|---|
| Top-3 retrieval relevance | 62.4% |
| Useful answer | 88.6% |
| Grounded answer | 84.1% |
Two findings mattered. First, the system was already far more useful than the retrieval number implied - a single relevant hit in the top-3 gave a 71% chance of a good answer, and pushing retrieval higher was hitting diminishing returns. Second, and this is the one I acted on: the biggest live weakness was groundedness. About one answer in six that got produced was not fully supported. When I traced those, almost all of them shared a shape: a long raw transcript was retrieved, the delivery layer truncated the chunk to a fixed character budget, the passage got cut off mid-sentence right before the load-bearing fact, and the model completed it from imagination.
The problem was the shape of the stored content, not the retriever.
The idea: distill the corpus into claims
Instead of storing raw transcripts and hoping the right slice survives truncation, I distilled each source into atomic claims:
- one claim per line, each self-contained (names its entities, never “it” or “the server”)
- tagged with a kind (decision, fact, config, howto, incident, status, preference) and a date
- proper nouns, numbers, IDs, paths and quoted strings kept verbatim
- ruthless compression: drop greetings, tool-call mechanics, dead ends that taught nothing; keep every decision and its reasoning
A 15KB rambling session becomes a dozen dense, dated, standalone statements. Each one fits comfortably inside the delivery budget, so nothing gets cut off mid-fact. The raw source still goes to cold storage for provenance and rollback; only the claims get indexed.
This is deliberately lossy. The bet is that what it drops (filler, redundancy, conversational scaffolding) is worth less than what it protects (facts that no longer get truncated).
Which model can be trusted to distill?
Distillation is a rewriting step, and a rewriting step can lie: invent a number, mangle an ID, drop the one fact that mattered, or pad thin content into bloat. So before trusting any model, I ran an adversarial fidelity gate: distill the same sources with each candidate, then have independent auditors compare every claim against the raw source and count four failure modes - invented, mangled, dropped-load-bearing, and bloat.
| Engine | Invented | Mangled | Density vs baseline | Verdict |
|---|---|---|---|---|
| Small frontier model (fleet) | 0 | 0 | 1.0x | PASS (reference) |
| Large frontier model | 0 | 0 | 1.1x | PASS |
| Open 70B (in-house) | 5 | 3 | ~4x | FAIL |
| Open 30B MoE | 1 | 1 | ~5x | MARGINAL |
| Low-cost API, v1 prompt | ~1 | 0 | ~2.4x | MARGINAL (bloat + one verbatim secret leak) |
| Low-cost API, hardened prompt | 0 | 0 | ~1.1x | PASS |
Two lessons here. The open 70B model failed outright: it fabricated and it garbled tokens, which is the one thing a knowledge base can never tolerate. And the low-cost API was a MARGINAL-to-PASS story that only turned after I hardened the prompt: the first pass over-split everything into bloat, occasionally denied that content existed, and once copied an API key verbatim into a “claim”. Adding explicit rules - never reproduce a credential, merge related facts, never assert absence, copy dates digit-for-digit - moved it to a clean PASS. The prompt, not the model tier, was the lever. A cheap model with a disciplined prompt beat an expensive model with a loose one.
The result: answers got better, and got more grounded
After distilling the full corpus and re-running the same 132-question evaluation on the live system:
| Metric | Before (raw) | After (distilled) | Change |
|---|---|---|---|
| Useful | 88.6% | 93.9% | +5.3 |
| Grounded | 84.1% | 93.9% | +9.8 |
Both changes are statistically significant (McNemar exact test, p=0.016 for useful, p=0.024 for grounded). The mid-truncation failure mode that produced most of the hallucinations disappeared entirely: because claims are short and self-contained, there was nothing left to cut off mid-fact. This is the part I want to underline: distillation is usually assumed to trade groundedness away for brevity. Here groundedness went up, because the mechanism that caused ungrounded answers was truncation, and distillation removed it.
I then repeated the experiment on a second, larger private corpus with a paired A/B design (same source, same question, answer built from the distilled claims versus from the truncated raw). The lift was even larger there: usefulness +18.8 points (72.5% to 91.3%) and groundedness +21.3 points (75.0% to 96.3%), both significant (p=0.003 and p<0.001, n=80). That corpus had more long structured documents, so it had more truncation to lose - which is exactly the shape the mechanism predicts.
A side benefit I did not optimize for: the indexed corpus got roughly 12x smaller. That does not automatically translate into cheaper search (the delivery budget caps that separately), but it does mean faster embedding and no truncation.
Making it permanent: distill-on-ingest
A one-time backfill decays: every new conversation and note lands raw and slowly re-dilutes the corpus. So the property has to be maintained, not achieved once.
- On write: when a new session is archived, it is distilled at ingest time by a local model call before it is indexed. If distillation returns nothing usable (junk, too short, an engine hiccup), it falls back to indexing the raw text - distillation never blocks the archive.
- As a safety net: a daily job sweeps every workspace for anything that landed raw (hand-written notes, external crawls, stragglers) and distills it. It runs in CI, not on my laptop, so it works whether or not my machine is on.
The engine choice for the standing pipeline came down to a simple realization: on-ingest is incremental, a handful of sources at a time, so the per-source cost of a good model is trivial. There was no reason to reach for the cheapest option and its privacy trade-offs when the volume was small. The bulk one-time backfill is where cost matters; the steady state is not.
Honest caveats
- The evaluation judges with the same model family that could be answering. It is an LLM-as-judge setup without human-written gold answers. The before/after comparison is apples-to-apples and the significance is real, but the absolute numbers should be read as directional, not as ground truth.
- The second corpus used synthesized questions. I had no real query log for it, so the questions were generated from the sources. That is fair for a paired A/B (both arms see the same question) but it inflates absolute scores; only the difference between arms is trustworthy.
- Distillation is lossy by design. It drops nuance - the exact wording of a debate, a reasoning chain that got superseded. For a lookup-oriented knowledge base that is the right trade. For a corpus where the raw phrasing itself is the value, it would be the wrong one.
- Engine quality varies and must be gated, not assumed. A result proven with one model does not transfer to another. Every engine that touched the corpus had to pass the fidelity gate first, and one popular open model failed it.
What I would tell someone starting
If your RAG feels stuck, measure at the answer level before you tune the retriever - the two can diverge by 15-25 points, and you might be optimizing the wrong one. And if your answers are ungrounded, look at what your delivery layer does to long sources before you blame the model. In my case the single highest-leverage change was not a better embedding, a better reranker, or a bigger context window. It was rewriting the corpus into claims short enough that the truth never got cut off.