Validation methodology: a primer
How ModelGenius decides whether a model switch is safe — and why you can trust the answer.
The question we actually answer
Model benchmarks tell you how models perform on someone else's tasks. They can't tell you whether your application — with your prompts, your output formats, your edge cases — will keep working if you swap the model underneath it. That is the only question that matters when real traffic is on the line, and it's the one a ModelGenius validation is built to answer: is it safe to switch, and is it worth it?
We treat this as a regression question, not a leaderboard question. The candidate model doesn't need to be impressive in the abstract. It needs to do what your current model already does, at least as well, on the work you actually send it.
Replay, not benchmarks
A validation starts from snapshots: captured request/response pairs from your production traffic (collected with our open-source capture SDK, or exported from logs you already have). We replay the same requests — same prompts, same parameters, same tools and schemas — against both your current model (the baseline) and each candidate.
Because both models answer the identical requests, every difference we measure is attributable to the model, not to a change in the task. Your baseline's own behavior is the yardstick throughout: candidates are compared against what your application gets today, never against an abstract ideal.
Success criteria are defined before any scoring
Before a single output is judged, we derive success criteria from your workflows: what each call is supposed to produce, which fields must be present, what counts, limits, or formats the downstream code depends on, and what "good" means for the free-text parts. Each criterion has an explicit definition of pass, fail, and — importantly — inconclusive, for outputs where a verdict genuinely can't be determined.
Fixing the criteria up front matters: it means the goalposts can't move after we see how a candidate performs, and it means every verdict in your report traces back to a named, inspectable criterion rather than an overall vibe.
Three kinds of checks
Different requirements deserve different measurement tools, so each criterion is evaluated in the mode that fits it:
- Deterministic structural checks. Is the output parseable JSON? Does it match the declared schema? Are required fields present and correctly typed? These run as exact, repeatable code — no judgment involved.
- Programmatic criterion checks. Requirements that are objectively verifiable ("exactly three items", "every score between 0 and 100", "no duplicate IDs") are evaluated by purpose-built check scripts, so the result is countable, not opinion.
- Rubric-based judging. Qualities that need reading — did the response follow the instructions, is the reasoning sound, is the tone right — are scored by an LLM judge against defined rubrics. We don't take judge models on faith: our judging pipeline is itself calibrated against human-certified reference labels, and we track its agreement rate the same way we'd track any instrument.
Counting honestly
Every criterion produces pass rates for baseline and candidate, and the number we care about is the delta between them. Two things keep that number honest:
- Confidence intervals. A candidate that passes 19 of 20 samples is not "95% reliable" — with so few samples, the plausible range is wide. We report Wilson score intervals on every rate, so you see not just the estimate but how much the sample size actually supports. Zero observed failures doesn't mean zero failure rate, and the report never pretends otherwise.
- Enough samples to matter. Confidence grows with evidence. A validation runs on enough of your real traffic for the intervals to become decision-grade, and the report shows the confidence achieved, not just a thumbs-up.
Prioritization and early stopping
Not all failures are equal. Criteria carry priorities that reflect what breaks your product versus what merely differs, and a validation run monitors high-priority regressions as evidence accumulates — pausing early when a serious regression is already clear, rather than spending your budget confirming a lost cause.
The verdict
A completed validation lands on one of four recommendations:
- Recommended — the candidate is comparable or better on your criteria, with a meaningful cost or latency advantage.
- Safe to switch — no meaningful regressions at the confidence achieved.
- Safe to switch (with fixes) — the only failures are mechanical: known API compatibility quirks, or formatting deviations that a small, deterministic compatibility fix resolves. We verify each fix against your actual outputs and ship it with the report, so "with fixes" means with these specific fixes, already tested — not "fixable in principle."
- Not safe to switch — real regressions on criteria that matter. The report shows exactly which ones, with examples.
What this isn't
No leaderboards, no synthetic test sets standing in for your workload, no training on your data. And no black box: every number in a ModelGenius report decomposes into named criteria, per-sample verdicts, and the checks that produced them — so your team can audit the reasoning, not just the conclusion.
Questions about the methodology? Contact us — a real person will answer.