I increasingly think RAG projects have a strange habit: whenever the system gives a wrong answer, the first suspect is always the model.
This is understandable. The model is the most visible part of the system, and it is also the easiest part to replace. Swap in a new model and the output changes immediately. Change a prompt and you can find a few better answers the same day. Compared with that, cleaning documents, checking versions, and designing tests all feel slow and unglamorous. They rarely create a dramatic demo moment. Most of the time, they simply make certain mistakes stop happening.
But the quality of a production system is defined exactly by those mistakes that no longer happen.
If I had to describe the work with a deliberately exaggerated ratio, I would say a RAG project is about 50% evaluation, 40% data work, 8% integration with the business, and only 2% model training or fine-tuning. This is obviously not an industry statistic. The point is that where a team spends its attention, and where the outcome is actually determined, are often not the same place.
We like optimizing models perhaps not because the model is always the most important part, but because it gives the strongest feeling of progress.
Evaluation does not give that kind of instant reward. The first thing it does is not even to improve the system. It destroys a comfortable ambiguity: what does “the answer is pretty good” actually mean?
Does it mean the system found a relevant document? Does it mean it cited a real page? Or does it mean the answer used the currently valid policy, preserved the original constraints, and admitted uncertainty when the material was not enough?
If these questions are not answered in advance, every optimization is driven by feeling. The team tests a few familiar questions again and again. A new model happens to answer them more smoothly, so it feels better. Whether it regressed on other questions, whether it is more likely to present old material as a new conclusion, or whether it merely expresses the wrong answer more convincingly, nobody really knows.
Optimization without evaluation often ends up optimizing not the system, but the team’s impression of the system. Useful evaluation is not just about giving an answer a score. A score tells you that the result changed. It does not tell you why it changed.
An answer can be wrong because the right material was never retrieved. It can be wrong because the right material was retrieved together with an expired version. It can be wrong because the source document was fine, but a table and its footnotes were separated during processing. It can even be wrong when all the evidence is present, because the model turned “usually allowed” into “always allowed.”
On the surface, all of these look like the same problem: the answer is inaccurate. In reality, they require completely different fixes. If you do not know where the error happened, changing the model is like a doctor giving every patient the same expensive medicine without first making a diagnosis.
Imagine an employee asks, “I’m traveling to Shanghai next week. What is the maximum hotel reimbursement?”
The knowledge base contains three documents: an old policy from 2024, a new policy from 2026, and a supplemental notice that applies only to the sales team. The title of the old policy is closest to the employee’s question, so the system ranks it first. The model reads it and answers, “800 yuan per night,” with a citation that is completely real.
This is the most dangerous kind of wrong answer, because it does not look wrong. The writing is fluent. The number is accurate. The citation opens. Unless someone continues to ask whether this policy is still in effect, which department the employee belongs to, and which reimbursement tier applies to Shanghai, the system appears to have done its job.
But the answer was already wrong long before the model wrote “800 yuan.”
The mistake began when the three documents entered the system.
The system treated the 2024 policy and the 2026 policy as two similar pieces of text, instead of understanding them as two successive states of the same rule. It knew both documents were about reimbursement, but it did not know the latter replaced the former. It also knew the sales notice contained a number, but it did not know that number only applied to a specific group of people.
From the perspective of text, all three documents are relevant. From the perspective of the business, they cannot be treated as equals.
This is why the phrase “data cleaning” can be misleading. It sounds like deleting duplicates, fixing garbled characters, and normalizing formats. Those things matter, of course. But the hard part is not merely making the text cleaner. The hard part is deciding what each document means inside this knowledge system.
A policy is not just a block of text. It is a rule issued by a certain department, at a certain time, for a certain group of people, and it may later be replaced by another rule. A table is not just a sequence of numbers. It may be the basis for a conclusion, and its header, footnotes, and surrounding section all constrain how those numbers should be understood.
When we cut documents into disconnected chunks, the sentences often remain, but their identity can disappear. The model may still see “the hotel limit is 1,200 yuan,” while losing the fact that this line belongs to “overseas travel,” “senior employees,” and a “trial policy.” It has not misread a single word, and yet it can still give the wrong answer.
This is why I increasingly think of data work as organizing the state of knowledge. We need to know who issued a document, when it became effective, when it expired, whom it applies to, and how it relates to other documents. Sometimes people call this ontology, but the word is less mysterious than it sounds. It simply forces us to answer a basic question: what are the important things in this system, and which differences between them must not be lost?
If the system has no concept of “replaces,” then old and new policies are just two similar files. If it has no concept of “scope,” then a sales-team rule can be answered to everyone. If it has no concept of “effective date,” the retrieval system can only guess which document deserves to be trusted.
A model cannot reliably recover information the system never preserved. This is also why a larger context window does not automatically solve the problem. Showing the model more text is not the same as telling it where that text sits in the real world. Putting more fragments on the table does not turn them back into a complete file.
At this point, evaluation and data work are no longer two separate activities.
Evaluation exposes which distinctions the system is missing. If tests repeatedly show that old policies are being used as current policies, we learn that “effective” and “expired” must become explicit information. If answers for different departments keep getting mixed together, we learn that “scope” cannot remain buried in the body text.
The reverse is also true. Whenever the knowledge structure gains a new distinction, the test set needs new questions to check whether the system actually uses it.
So 50% evaluation and 40% data work are not two preparation stages that happen before the model work begins. They are more like a pair of gears turning together. Errors reveal what the system does not understand. New structure gives the system a way to understand it. New tests verify whether that understanding is real.
This also explains why many RAG projects look good in demos and then get worse after launch. A demo faces a static collection of files. Production faces an organization that keeps changing. Policies get updated. Departments are reorganized. Products are renamed. Labels that were once correct slowly become wrong.
Data is not fuel you prepare once and burn. It is the reality the system has to keep maintaining.
Business integration matters for the same reason. Access control, updates, and logs are often treated as peripheral engineering, but they decide whether this reality is trustworthy.
Access control determines whether a piece of knowledge exists for the current user at all. Updates determine whether the system is answering today’s question or last month’s question. Retrieval records determine whether, after a wrong answer appears, we can walk back along the original path and see why the system selected that material.
Without this, the team can only stare at a wrong answer and guess. Maybe it was the model. Maybe it was retrieval. Maybe it was the data. Everyone can offer an explanation, but nobody can reproduce the failure. A system like that is hard to improve because it has no memory of how it failed.
A healthier process looks different. An error appears online. We can return to the exact material used at that moment, identify which piece of information was missing, change the document relationship or answering rule, and then rerun all the previous questions to make sure the fix did not break something else.
The system does not become reliable through one heroic upgrade. It becomes reliable because every failure leaves behind something the team can learn from.
The model still matters. It just should enter the conversation later.
When the right material is consistently retrieved, versions and scopes are preserved, the question contains enough information, and the model still repeatedly misunderstands complex clauses, reads tables incorrectly, or fails to combine multiple pieces of evidence, then we can say the model is the bottleneck. At that point, training has a clear target. The goal is not vaguely to “make the model understand company knowledge better,” but to fix a stable, repeated behavior that can be tested.
Facts that change often should usually not be trained into the model at all. Reimbursement amounts, product prices, and current policies all change. Once these facts are hidden inside the model, they become harder to update and harder to explain. A more robust approach is to keep facts outside the model, where they can be updated and inspected. The model should read, compare, and express them. It should not secretly become the only copy of the company’s rules.
This is also the view we gradually developed while building Knowhere. At first, the problem looked like document parsing. Later, it became clear that what we needed to preserve was not just text, but the world around the document: where sections begin and end, which explanation a table belongs to, how images connect to nearby text, where a piece of material came from, and along what path an answer found it.
Knowhere cannot guarantee that every answer is correct. No document component can do that by itself. What it can do is avoid destroying, at the moment a document enters the system, the context that will later be needed to judge correctness. Then, when an answer is disputed, a person or another layer of the system still has something to inspect, instead of being left with a piece of text that has been detached from its source.
I no longer judge the maturity of a RAG project mainly by which model it uses. I care more about a different set of questions. Can the team clearly say what counts as wrong? Can a failure be traced back to a specific cause? Have document versions, structure, and scope been preserved? When the evidence is not enough, is the system actually willing to stop?
A model that answers questions is easy to get excited about. But a reliable knowledge system needs a quieter capability: it must know not only how to produce an answer, but also which answers should never be allowed through.
Originally published on Medium.
Blog 