The demo works. Production is a different problem.
LangChain makes the first version of an LLM feature cheap to build. What it does not give you is retrieval over your own records, a way to see a regression, a cost you can predict, or a boundary that keeps customer data out of a model that should never have seen it. We build that part.
The answers got worse and no test caught it.
The feature shipped and the answers were good. Then a prompt was edited, a model version moved underneath it, a document was reindexed, and the answers got worse in a way no test caught. Without evaluation and tracing, quality is whatever the last complaint says it is.
Grounded, traced, budgeted.
LangChainWhat production actually needs.
Retrieval over your own records
Chunking, embedding and reranking tuned against the documents the business actually asks about, not a generic corpus. Answers cite the record they came from, so a wrong answer is traceable to a wrong source.
Evaluation before anyone ships
A held-out set of real questions with graded answers, run on every prompt and model change. A regression shows up as a failed run rather than as a support ticket a fortnight later.
Tracing on every call
Each run carries the retrieved context, the resolved prompt, the tool calls and the tokens spent. When an answer is disputed, someone can open that exact run instead of guessing.
Cost and latency budgets
A per-request ceiling on tokens and time, with caching, smaller models on the easy path and a hard timeout. A feature whose worst request costs orders of magnitude more than its typical one is a feature finance will switch off.
Prompts under version control
Prompts, retrieval settings and model versions live in the repository and ship through the same review as the code. Nobody edits production behaviour in a text box.
The data boundary
What is allowed to enter the context window, what is redacted, what is retained and by which provider, decided per data class before the first call. Retrieval respects the permissions of the person asking.
Scope, ground, measure, harden.
Scope
One task the feature has to do, with the answers a good version gives and the answers it must refuse. Written down before the framework is opened.
Ground
The sources, their permissions and their refresh cadence, then retrieval built against them and read back by someone who knows the documents.
Measure
The evaluation set, the graders and the cost and latency budget, wired into the build so a change that breaks any of them fails there.
Harden
Rate limits, fallbacks, redaction and retention settled, and the traces handed to the team that will run it on a normal Tuesday.
The retrieval is wrong long before the model is.
Most bad answers are not a reasoning failure. The right document was never indexed, was chunked through the middle of the table that held the answer, or was returned to someone who should not have been able to see it. Fix retrieval and permissions first; the model is rarely the expensive part.
Tell us what the demo could not survive.
Tell us what you are running
What the system does today, where it breaks, and when it has to work. An engineer reads it — you get an answer inside one business day, not a sequence.