Evaluating LLM Features: Evals, Not Vibes

The demo always works. That is the trap. You wire up an LLM feature, you try five or six prompts you thought of at your desk, the answers come back looking sharp, and you ship. It feels responsible because you tested it. You did not test it. You looked at it. Those are different things, and the gap between them is where most AI features quietly fail once real users start sending real inputs.

I have shipped enough of these to know the pattern. A model looks great on the inputs you imagined and falls apart on the inputs people actually send, which are stranger, longer, and more adversarial than anything you would think to type. Worse, the moment you change a prompt or a provider ships a “better” model, the behavior you liked can shift underneath you, and you have no way to know whether you improved things or broke them. Vibes cannot answer that question. Evals can.

This is the discipline I want to lay out: how to evaluate LLM features rigorously instead of on gut feel. What an eval actually is, how to build a set worth trusting, the three families of scoring methods and where each one lies to you, the difference between offline evals and online outcomes, and how to make all of it run automatically so every change is measured. None of it is glamorous. All of it is the work.

Why “it looked good in the demo” is not evidence

Traditional software is deterministic. Same input, same output, and when that breaks you call it a bug and fix it. LLM features are probabilistic. The same prompt can return two different answers on two calls, both defensible, and neither is a bug. That single fact dismantles the way most teams think about testing, because you cannot assert an exact value against a system that does not produce exact values.

So the demo proves almost nothing. Five prompts sampled from your own imagination tell you how the model behaves on a tiny, biased slice of the input space that happens to look like what you expected. It says nothing about the customer who pastes a 4,000-word email and asks for a summary, or the one who writes in three languages in the same message, or the one who is actively trying to get your feature to say something embarrassing. Those are the inputs that decide whether the feature works, and the demo never contains them.

There is also the regression problem, which is the one that bites teams who have shipped and feel safe. You tweak the system prompt to fix one annoying case. It fixes that case and silently degrades four others you were not looking at. Nobody notices until a user complains, and by then you cannot even reproduce the old behavior. Without a repeatable measurement, every change is a blind swap. This is the same reason building AI-native products is a genuinely different discipline: the thing you are shipping generates its own variability, and you have to measure your way through it rather than reason your way through it.

What an eval actually is

Strip away the jargon and an eval is two things: a repeatable test set and a scoring method. That is it. A set of inputs you care about, paired with a way to decide, for each output, whether the model did well or badly. Run the set, get a score, and now a change to your prompt or model produces a number you can compare against the number from before.

The comparison is the entire point. An eval is not about proving the model is “good” in the abstract. It is about telling you, reliably, whether version B is better or worse than version A on the cases you have decided matter. That reframing matters because it lowers the bar to starting. You do not need a perfect, comprehensive benchmark before an eval is useful. You need a set that is representative enough that moving its score correlates with moving real quality, and a scoring method consistent enough that the same output gets the same grade twice.

I lean on my A/B testing background here, because the logic is identical. In experimentation you never trust a single observation, you define a metric, gather a sample, and compare variants against it. An eval is the offline version of that same instinct applied to model quality. Measure, do not guess. If you already run a serious A/B testing program, you already have the mental model; you are just pointing it at model outputs instead of button colors.

Building an eval set that reflects reality

The eval set is where the quality of the whole exercise is decided, and it is where teams cut the most corners. A set of ten cases you invented is barely better than the demo. A set that mirrors what users actually do is the difference between a score you can trust and a number that flatters you.

Start with real inputs. Once you have any usage at all, your logs are the richest source you will ever get: the actual prompts, the actual documents, the actual messy phrasing real people used. Pull a representative sample across the range of what you see, including the boring common cases, because those are the bulk of your traffic and they are what a regression will hurt most. If you have not shipped yet, get as close as you can with realistic hand-written cases, then replace them with real ones the moment traffic exists.

Then add adversarial inputs on purpose. The cases that break things are rarely the average ones; they are the extremes. The empty input. The input in the wrong language. The prompt injection attempt buried in a pasted document. The question that sits right on the edge of your safety policy. The absurdly long input that blows past what you tested. You are not being paranoid, you are covering the space users will find whether you look or not. A good eval set is a mix: mostly representative of typical usage, with a deliberate tail of adversarial and edge cases that each encode a failure you never want to see again. And every time production throws a genuinely bad output, that case goes into the set. The eval set should grow scars over time, one per incident.

Scoring method one: programmatic and exact checks

The cheapest, most reliable scoring you can do is code. Wherever a correct answer has a checkable property, write an assertion and let a machine grade it deterministically. This is the scoring method you should always reach for first, because it is fast, free, and never disagrees with itself.

Plenty of LLM output has structure you can verify without judgment. Did it return valid JSON? Does the JSON match the schema? Is the required field present? Did it pick one of the allowed enum values? Does the extracted date parse? Is the classification label in your known set? For anything with a single right answer, exact or fuzzy string match works. For anything structured, schema validation works. These checks cost nothing to run a thousand times and they catch the failures that embarrass you most, like a downstream parser crashing because the model wrapped its JSON in a friendly sentence.

The limit is obvious: programmatic checks only measure form, not quality. They can confirm a summary is valid JSON of the right shape; they cannot tell you the summary is any good. So you use them as the first gate, the cheap filter that catches format and safety violations before you spend money on anything more expensive, and then you reach for a method that can judge substance. The discipline of forcing structured, checkable output in the first place is part of prompt engineering for production, and it pays off directly here because it expands how much you can grade with code alone.

Scoring method two: human review

For quality that resists a formula, humans are the ground truth. A person reads the output and rates it, against a rubric, and that judgment is the standard everything else is measured against. When you want to know whether a summary is actually faithful, whether a tone is right, whether an answer is genuinely helpful, a careful human reviewer is the most trustworthy signal you have.

The catch is that human review does not scale and it is not as consistent as people assume. It is slow and expensive, so you cannot run it on every change, and two reviewers will grade the same borderline output differently unless you pin them down. The fix is a rubric: explicit, concrete criteria that turn “is this good” into a set of specific questions with defined answers. Score faithfulness on a defined scale with examples of each level. Say exactly what counts as a format failure. The more you remove interpretation, the more your human scores mean, and the more they can serve as the gold set you validate cheaper methods against.

This is precisely the work I do with Micro1.ai, designing how reasoning quality gets scored for LLM-training work. Building a scoring scheme that different graders apply consistently is genuinely hard, and it taught me that the rubric is the actual product. A vague rubric produces noisy scores that feel rigorous and measure nothing. A sharp rubric, with worked examples of a 1 versus a 3 versus a 5, produces scores you can bet a release on. Most teams underinvest here and then wonder why their human evals are so inconsistent; the answer is almost always the rubric, not the reviewers.

Scoring method three: LLM-as-judge, and its traps

You cannot afford humans on every run, and code cannot grade substance, so the practical middle path is using a model to grade another model’s output. LLM-as-judge scales like code and can assess quality like a human, at least roughly. Give a judge model the input, the output, and a rubric, ask it to score, and you have something you can run on thousands of cases for the price of some tokens. Used well, it is the workhorse of a real eval program.

Used carelessly, it lies to you in specific, well-documented ways, and you need to know them. Judges have biases: they tend to prefer longer answers, they favor outputs in their own style, and in a head-to-head they often prefer whichever response was shown first regardless of quality. They can be inconsistent, grading the same output differently across runs. And they will confidently score things they cannot actually verify, like factual accuracy on a topic they know nothing about.

The worst version of this trap is letting a model grade its own homework. If the same model family both generates the output and judges it, you have built a system inclined to approve of itself, and your scores drift toward flattery. There are ways to keep the judge honest: use a strong, capable model as the judge rather than the cheapest one; give it a tight rubric instead of an open-ended “rate this”; ask for a structured verdict with reasoning so you can audit why it scored the way it did; and, the non-negotiable one, validate the judge against a human-graded gold set before you trust it. If the judge does not agree with your careful human scores on a sample, it is not measuring what you think, and you fix the rubric or the judge before you rely on the number. A judge is a measuring instrument, and you calibrate an instrument against a known standard before you ship readings from it.

Offline evals versus online outcomes

Everything so far is offline: a fixed set, scored in a controlled run, before anything reaches a user. Offline evals are how you move fast and change things safely, because they are repeatable and cheap and they gate your releases. But a high offline score is a proxy, not the goal. The goal is that the feature does something good for a real person, and only production tells you that.

So you close the loop with online metrics. Offline you might measure faithfulness and format adherence on your set; online you watch whether users accept the output, edit it heavily, regenerate it, thumbs it down, or abandon the flow entirely. The two are meant to reinforce each other. When online metrics dip, you mine the bad sessions, turn them into offline eval cases, and now your set is a truer proxy for what users experience. When your offline score climbs but the online metric does not move, your set is measuring the wrong thing and needs to change. The relationship between proxy and outcome is exactly the discipline of hypothesis-driven experimentation: you state what you expect a change to do to a real metric, ship it behind a test, and let the numbers confirm or kill the belief. Offline evals let you iterate quickly; online experiments tell you whether the iteration mattered.

What to actually measure

“Quality” is not one number. A useful eval program tracks a handful of dimensions, because a feature can be excellent on one and unacceptable on another, and an average hides that.

Faithfulness and hallucination come first for anything grounded. If the feature answers from retrieved context, you measure whether the answer is supported by that context or invented, because a confident fabrication is the failure that destroys trust fastest. In practice this means checking claims against the source rather than against the world, which is one reason retrieval quality is so tied to eval quality; you cannot fairly grade faithfulness if you do not also know whether the right context was retrieved in the first place. Format adherence is next, and it is cheap to measure and expensive to ignore, since a malformed output breaks whatever consumes it. Safety is its own axis: refusal rates on legitimate requests, and, more importantly, harmful compliance on the requests you built your set to catch.

Then the two everyone forgets until the bill arrives: latency and cost. These are not infrastructure footnotes, they are quality dimensions users and finance both feel. A more accurate answer that takes eight seconds may be worse product than a slightly weaker one that returns instantly, and a prompt change that quietly triples token usage can be a regression even if the outputs improve. Track them in the same eval run as everything else, so a “better” prompt that doubles cost shows up as the trade-off it is, not a surprise next month.

Make evals part of CI

An eval you run manually when you remember is an eval you will stop running the week you get busy. The whole thing only works when it is automatic, when every change to a prompt, a model, or the retrieval logic triggers the eval set and reports the scores before the change can merge. Wire it into CI the same way you wire in your test suite, because for an AI feature it is your test suite.

The payoff is that regressions get caught by a machine instead of a customer. Someone opens a pull request that tweaks the system prompt; the eval runs; the faithfulness score drops two points and the format failures tick up; the pull request is blocked with the exact cases that broke. That is the loop that lets a team change AI behavior with confidence instead of fear. It also disarms the most dangerous event in an LLM product, the model upgrade. A provider ships a new version and deprecates the old one, and “newer” does not guarantee “better for you”; it can be better on average and worse on your specific cases. With evals in CI, a model swap is a candidate you run the set against and compare, not a leap of faith. Without them, you find out from your users. Treating a model upgrade as a change that must pass evals before it ships is core to shipping AI features safely, and it is the single habit that separates teams who sleep through model releases from teams who firefight them.

The mistakes I see most

The same failures repeat across almost every team I talk to, and they are worth naming plainly. No eval set at all, shipping on demos and hoping, which is where everyone starts and where too many stay. An eval set that is too small, a dozen cases that give a number precise enough to feel scientific and too thin to mean anything. Judging on vibes dressed up as rigor, where the “eval” is one person skimming a few outputs and calling it measured.

Then the subtler ones. Letting a model grade its own homework and trusting the flattering scores that result. Building the set once and never growing it, so it goes stale as your product and your users move. Measuring only accuracy and getting blindsided by cost, latency, or a safety failure you never scored. And the expensive one, not re-running evals on model upgrades, so a provider’s “improvement” silently breaks a case you cared about and you learn it from a support ticket. Every one of these traces back to the same root: treating evaluation as a thing you do once to feel safe, rather than a standing measurement system you maintain like any other part of the product.

The short version

• The demo is not evidence. Looking at a few outputs you imagined tells you nothing about the messy, adversarial inputs real users send.
• An eval is two things: a repeatable test set and a scoring method. Its job is to tell you whether version B beats version A on cases you care about.
• Build the set from real logs plus deliberate adversarial and edge cases, and grow it by one case every time production produces a bad output.
• Use three scoring methods in order of cost: programmatic checks for form, LLM-as-judge for scaled quality, and human review as the ground truth you calibrate the judge against.
• Never let a model grade its own homework, and validate any judge against human scores before you trust its numbers.
• Offline evals let you iterate fast and gate releases; online metrics tell you whether the iteration helped a real user. Feed bad sessions back into the set.
• Measure more than accuracy: faithfulness, format adherence, safety, latency, and cost, because a feature can pass one and fail another.
• Put evals in CI so every prompt change and every model upgrade is measured automatically, before it reaches anyone.
• The common mistakes are all the same mistake: treating evaluation as a one-time reassurance instead of a standing measurement system.

---

I am Deepanshu Grover, a Growth Product Manager and AI builder in Paris. If you are shipping AI features on gut feel, connect on LinkedIn or get in touch.