← All resources
IntelligenceDecember 3, 2025·8 min read

RAG that actually ships

An evaluation-first approach to retrieval-augmented features that survive contact with real users.

By Axicial

Most RAG demos work. That's the problem. A retrieval-augmented feature can look flawless in a demo with five hand-picked questions and then fall apart in week two of production, because the demo tested the model's ability to sound confident, not the system's ability to be right. The gap between those two things is where most AI features quietly die.

Start with the eval set, not the prompt

The single biggest mistake teams make is writing the prompt first and the evaluation harness never. We flip that order. Before a line of retrieval code gets written, we build a set of real, adversarial queries — pulled from actual support tickets, actual documents, actual edge cases — and hand-label what a correct answer looks like, including "the system should say it doesn't know." That last category matters more than any other, because an unhedged wrong answer is worse than no answer at all, and it's the failure mode that never shows up in a happy-path demo.

On one lending-adjacent system we built, this meant hand-scoring roughly 150 queries against real financial documents before we picked a single embedding model. That sounds slow. It saved us from shipping a chunking strategy that looked fine on paper and silently dropped numeric fields — tables and line items — from every retrieved chunk, which is exactly the kind of failure that only shows up when you're grading against ground truth instead of vibes.

Chunking is a data modeling problem, not a text-splitting one

Fixed-size chunking (500 tokens, 50-token overlap, repeat) is the default in every tutorial and it's wrong for almost anything that isn't prose. Financial statements, contracts, structured forms — these have a shape, and the shape carries meaning. A line item split across two chunks is a line item your retriever can't reason about. We chunk along document structure first (sections, tables, clauses) and only fall back to sliding windows for genuinely unstructured prose. It's more upfront engineering and it's the difference between retrieval@5 in the 60s and retrieval@5 in the 90s on the eval set.

Retrieval quality, not generation quality, is where systems fail

Teams spend weeks tuning prompts and hours on retrieval, and it should be the reverse. If the right chunk isn't in the context window, no amount of prompt engineering recovers it — the model is being asked to reason about information it was never given. We track retrieval metrics (precision@k, recall@k) as a separate, gated step before we even look at end-to-end answer quality. If retrieval is broken, fixing the prompt is polishing the wrong layer.

Reranking earns its cost here. A cheap bi-encoder retrieval pass followed by a cross-encoder rerank over the top 20-30 candidates consistently outperformed a single high-recall retrieval step in every system we've measured, at a latency cost that's usually invisible to the user if it's run async against the generation call.

Guardrails belong in the pipeline, not the prompt

"Please don't hallucinate" is not a guardrail, it's a request. The systems that hold up in production have structural constraints: generation is required to cite the source chunk it drew from, and a lightweight verification pass checks that cited claims actually appear in the cited source before the answer reaches the user. When that check fails, the system degrades to "I don't have enough information" rather than silently emitting an unsupported claim. This is slower and more expensive than trusting the model, and it's the single change that made stakeholders comfortable putting the feature in front of real customers instead of an internal beta forever.

Shipping means monitoring, not launching

The eval set from week one doesn't retire at launch — it becomes the regression suite. Every prompt change, every embedding model swap, every reranker tweak runs against it before it goes out, the same way you wouldn't ship a code change without running the test suite. On top of that, we sample a percentage of live traffic for ongoing human review, because production queries drift from whatever you anticipated at build time — new document types show up, users phrase things you didn't predict, and the eval set needs to grow with them or it stops meaning anything.

The honest version of "we built a RAG feature" is closer to "we built a retrieval system, an evaluation harness, a verification layer, and a monitoring loop, and somewhere in there is a language model." Treat it like the last piece of a system rather than the whole system, and it survives contact with real users. Treat it like the whole system, and it survives about two weeks.

Want this kind of work on your project?

Tell us what you’re building and we’ll map the fastest path from idea to launch.

Start a project →