Is Your AI Feature Actually Working? How to Measure AI Performance After Launch

Overview

A demo shows an AI feature can work once — it cannot prove the feature works reliably. This guide shows founders how to define success in writing, layer four practical checks, and turn every failure into a regression test.

Key Points

• A successful demo shows a feature can work once — it cannot establish that it works reliably. Evidence is a written definition of success, checked against real usage.
• Generative AI is variable: testing once before launch is not a strategy — measurement has to be ongoing.
• Layer four checks: human review of real outputs, deterministic checks, AI grading calibrated by humans, and production numbers including cost and speed.
• Turn every failure into a permanent test case — capability tests set the hill to climb, regression tests stop backsliding.
• Regression tests should stay near perfect — a dip signals a possible regression to investigate; capability tests should keep challenging the system. Beware vibes — keep reading raw outputs.

The short answer

A successful demo is limited evidence: it shows your AI feature can work once. It cannot establish that the feature works reliably in production. You know an AI feature is working when you have written down what “working” means and check that definition against real usage.

The failure mode looks like this: the feature ships, nobody complains for a while, and everyone assumes it is fine — until a customer forwards a confidently wrong AI answer, or the model bill arrives far above estimate. That is a measurement problem.

One note on scope: this guide is about LLM-powered features — assistants, triage tools, content generators — where outputs vary from run to run. If you are still deciding what to build first, our guide to AI solutions for startups covers the planning stage; this article picks up where that one leaves off, after launch.

Why AI breaks the “test it once” habit

Traditional software is mostly deterministic when its inputs, data, configuration, dependencies, and environment are fixed — so a passing test suite often keeps passing until something changes. But behaviour can still shift without a code change: data drifts, a setting is edited, an external service updates. Generative AI adds another layer of variability: models can produce different outputs from the same input, which makes traditional software testing methods insufficient for AI architectures — as OpenAI states in its evaluation guidance.

That variability has practical consequences: an assistant can answer well on Monday and badly on Friday; a small prompt change can fix one failure and introduce others; a new model version can shift behaviour without warning. If you only tested before launch, you are flying blind after it. Anthropic describes teams in this position as stuck in “reactive loops” — catching issues only in production, where each fix risks new failures.

There is a second trap: mistaking benchmark scores for evidence about your product. Benchmarks test general models on shared tasks; they say little about your feature's behaviour with your customers. Practitioner guidance draws the line: product evaluations — not model benchmarks — measure whether your product does what you want.

Step one: write down what “good” looks like

Before measuring anything, define success in writing, in plain language, for each AI feature. A useful definition has three parts: what the feature must get right, what it must never do, and what “good enough” means numerically. For example: a triage assistant must categorise requests correctly, must never invent a company policy, and should resolve a defined share of routine requests without human correction.

This is eval-driven development: write the definitions into tests before the system can fully satisfy them, then iterate. It also removes a common ambiguity: two people reading the same spec often imagine different edge cases. That is the clarity AI Solutions planning establishes before production investment — explicit success signals, agreed up front.

Four checks that together tell the truth

No single check catches everything. Layer several methods, each covering the others' blind spots — a “Swiss cheese” approach, as Anthropic's evaluation guide describes it. For a startup, four checks are enough to start.

1. Look at real outputs yourself, regularly

Read a random sample of real AI outputs in a short weekly session. Look for failure patterns: repeated wrong answers, confident inventions, slow responses, tasks the AI silently gave up on.

Practitioners who teach evaluation to product teams recommend starting with error analysis rather than infrastructure — about half an hour on a few dozen outputs after significant changes. Reading real outputs builds the judgement the other methods depend on.

A hypothetical illustration (no real company, outcomes, or figures implied): a property-management assistant files “no hot water” under general plumbing instead of flagging it urgent — the urgency rule was never written down. A summary dashboard is unlikely to surface the missing rule itself; reviewing individual requests could reveal the missing rule.

2. Write checks a computer can run

Some failures are exact and checkable without judgement: did the assistant call the right tool? Did the refund actually process, or did the AI merely say it did? Did the response arrive within the promised time? These deterministic checks are fast, cheap, and reproducible — use them for every behaviour with a right-or-wrong answer. Grade the outcome, not the precise path.

3. Let a second AI grade against your rubric — then check the grader

For qualities that need judgement — tone, completeness, whether an answer is grounded in your documents — a capable language model can score outputs against your step-one criteria.

But a grading AI needs the same supervision as the feature itself. Calibrate its scores against human judgement on a sample, give it an explicit way to say it lacks enough information rather than guessing, and prefer a different or stronger model than the one being graded. Human review keeps automated grading honest.

4. Track the numbers production actually produces

Quality is only part of the picture. From day one, track cost per task, response speed, error and escalation rates, and what users do after receiving an answer — on live traffic and on a fixed set of representative tasks. The fixed set answers whether a change made things worse. Cost and speed are part of whether the feature “works”: an assistant that is correct but far over budget has a problem, not a success.

Five numbers worth tracking

These five numbers make the trends visible. None has a universal “good” value — what counts is how each moves over time on your own traffic. This is a recommended starting set:

• Task success rate: if you assess success on a sample, divide successful evaluated tasks by all evaluated tasks in that sample. Define success in your step-one criteria first — for a triage assistant, a request counts when category and priority are both right. Grade a sample of real tasks against that definition, automating as checks prove reliable.
• Unsupported-answer rate: answers containing claims your sources do not support, divided by answers reviewed. Check answers against the documents and data they were allowed to use during the weekly review.
• Correction and escalation rates: calculate each separately over the same stated population and time window. Keep necessary escalation separate from avoidable failure — escalation designed in is a product choice; a task the AI should have handled but could not is a defect.
• Response time: how long users wait. Look beyond the average to the slowest responses too.
• Cost per successful task: divide costs and successful tasks from the same period, including spending on failed attempts. Include model and API usage, retrieval and search calls, tooling, and human review time.

Read these together: falling success with rising corrections points at quality; rising cost per success with steady quality points at economics.

Turn every failure into a test

The routine that compounds: every failure you find — in weekly reviews, complaints, or support tickets — becomes a permanent test case. Over time you accumulate two kinds of tests: capability tests (hard tasks the system should eventually handle — a hill to climb) and regression tests (tasks it already handles, where a dip signals a possible regression that needs investigation).

Anthropic's evaluation guide suggests twenty to fifty realistic tasks drawn from actual failures as a credible beginning. Grow the set as the product matures. Anthropic describes how evaluations can help accelerate development by making failures and behavioural changes visible. They also provide a baseline for investigating whether a change improved the system.

Traps worth avoiding

A few failure modes are worth naming explicitly.

Judging by vibes

“It seems like it's working” is not an evaluation strategy — OpenAI lists vibe-based evaluation as an explicit anti-pattern, alongside waiting until after launch to think about measurement. If nobody can point to the definition of good and the latest check, you are vibing.

Reading a perfect score wrong

A near-perfect regression score is exactly what you want — those tests guard behaviours the system used to handle. A dip signals a possible regression that needs investigation: account for run-to-run variation, grading problems, and test-environment changes before concluding that something broke. The caution applies to capability tests: Anthropic's guidance says capability evals should start at a low pass rate, targeting tasks the system struggles with. A capability set it passes completely no longer shows where to improve — it has saturated, so write harder tests.

Optimising the metric instead of the product

Once a number becomes the target, teams start gaming it — consciously or not. Keep reading raw outputs, and treat any metric that never moves as suspicious.

Ignoring cost until the invoice arrives

Model usage, retrieval, tooling, human review time, and maintenance when providers change interfaces all belong in the economics from the start. Track cost per task alongside quality.

What to do this month

If you have an AI feature live or launching, this is a recommended starting sequence:

• Write success criteria: what it must get right, what it must never do, what “good enough” means.
• Collect twenty to fifty realistic test cases from real usage and failures, including ones the system currently gets wrong.
• Hold a weekly thirty-minute review of a random sample of real outputs, with one quality owner.
• Track cost per task, response speed, and error rates from day one, on live traffic and your fixed test set.
• Re-run your checks after every meaningful change: prompts, model versions, tools, data sources.

Frequently asked questions

Is user feedback not enough?

It helps, but it is a weak foundation alone. Feedback is sparse, self-selected, and skewed toward severe failures. Treat user reports as new test cases, not as your measurement system.

We use an off-the-shelf AI tool rather than building our own. Do we still need this?

Yes, though the shape changes. You are not evaluating the underlying model — but your configuration: prompts, data, workflows, and behaviour with your customers. A generally strong model can still mishandle your tasks.

What will this cost us in effort?

That depends on the feature's complexity and traffic volume. Start with the routine above: write success criteria and test cases, then keep a short weekly review. Treat measurement as part of building the feature.

Sources

• Anthropic — Demystifying evals for AI agents (official engineering blog; published 9 Jan 2026; capability vs regression evals, eval saturation)
• Anthropic — Building effective agents (official engineering blog; published 19 Dec 2024; measure performance before adding complexity)
• OpenAI — Evaluation best practices (official API docs; eval workflow and anti-patterns; Evals platform read-only 31 Oct 2026, shutdown 30 Nov 2026)
• Hamel Husain — AI Evals: Everything You Need to Know (practitioner FAQ; product vs model evals; minimum-viable evaluation)

If your AI feature is already live and you are unsure it is holding up, start with the definition. iCodeLTD's AI Solutions service covers defining success signals before production investment — as part of adoption planning, while AI Development covers building the feature itself. If you are scoping an AI project, the project cost calculator gives an indicative budget and timeline range from scope and complexity. To talk through a live feature, use the contact page.