By Zahi Raifer
A team changes its model prompt and watches the success rate rise. Everyone is pleased until someone asks what the chart counts. It includes successful tool calls, but not requests rejected before a tool runs. It counts a three-call recovery sequence as three observations. Requests that never report a final status quietly disappear.
The chart is mathematically correct and operationally misleading.
Reliability measurement for AI tools starts before choosing a dashboard. It starts by defining the population being measured, the boundary of one attempt, and the evidence required to call that attempt successful.
Here is a practical framework for making those definitions explicit, without pretending that one percentage can describe the entire product.
Start with the user's unit of work
A model invocation is not necessarily a user task. One task may require several model invocations, several tool calls, and a final verification step.
For the website-editing category I work on at Orivox, a useful measurement question is whether one requested edit reached a verified outcome, not whether an individual model response looked plausible.
That is an example of metric design, not a report of product performance. The definitions below are proposed engineering practices rather than claims about measured results.
Assign a task identifier at the point where the application accepts responsibility for handling a request. Keep child invocation identifiers, but do not let their number silently redefine the task-level denominator.
Separate eligibility from execution
Some requests cannot proceed because they lack permission, target a missing page, or ask for an unsupported operation. These are real user experiences even when no model or tool executes.
Measure an acceptance funnel: received requests, eligible requests, admitted tasks, executed tasks, and verified completions. Define which transitions can fail and which component owns each event.
Then publish distinct rates. Completion among admitted tasks answers a different question from completion among all received requests. Both can be useful; neither should masquerade as the other.
Document exclusions in plain language. If a rate excludes unsupported requests, say so. A product that rejects more difficult tasks can improve its eligible-task success rate while helping fewer people overall.
Give every published metric a short definition document with an owner, numerator, denominator, exclusions, and version. When a definition changes, annotate the chart or start a new series. Otherwise a reporting change can look exactly like a product improvement, even when no user experience changed.
Count retries inside the task
Suppose ten tasks each need one successful tool call. Nine succeed immediately. The tenth fails four times and then succeeds.
The task completion rate is ten out of ten. The tool-call success rate is ten out of fourteen. The share of tasks completed without a retry is nine out of ten.
None of these numbers is wrong. They answer different questions about completion, infrastructure load, and first-pass behavior.
Show the distinctions rather than choosing whichever looks best. Also measure total task latency and resource consumption across the recovery sequence. A task that eventually completes after an exhausting wait is not equivalent to one that finishes promptly.
Make success an evidence claim
An assistant saying "done" is evidence about what the assistant said. It is not proof that the requested change happened.
Define a technical completion condition that can be checked independently: the intended resource changed, the persisted revision can be read back, and the relevant invariants still hold. The exact checks depend on the operation.
Separate that result from user satisfaction. A technically valid heading change may still misunderstand the requested tone. A person accepting the result provides another kind of evidence, with its own missingness and selection biases.
Avoid turning the absence of a complaint into a satisfaction score. Many dissatisfied users simply leave.
Keep unknown outcomes visible
A missing terminal event must not automatically become success, failure, or an excluded row.
It may represent a telemetry outage, a disconnected client, a task still running, or a genuinely lost operation. Give unknown outcomes their own count and age distribution.
Choose an observation window before comparing completion rates. A task accepted seconds before a report runs has had less opportunity to finish than one accepted yesterday. Group tasks by acceptance time, then evaluate mature cohorts after the agreed window.
For ongoing work, show pending tasks separately. For overdue unresolved work, show an explicit unknown or timeout classification based on documented rules. Do not repeatedly move difficult cases outside the reporting period.
Do not sample away the denominator
Detailed traces are expensive, so sampling them can be sensible. The small events needed to count accepted tasks and terminal outcomes should have a different reliability plan from verbose diagnostic traces.
If sampling is necessary for measurement events, preserve sampling probabilities and use an appropriate estimator. A sample biased toward errors cannot directly estimate the overall failure rate without correction.
Check telemetry coverage as a metric in its own right. Compare independently recorded admissions with outcomes, and alert when their relationship changes unexpectedly.
A sudden improvement in reported reliability during an event-delivery incident should trigger suspicion, not celebration. Sometimes the dashboard improves because the failures stopped reaching it.
Compare like with like
An overall rate can change because the workload changed rather than because the system improved.
Separate cohorts by a few meaningful dimensions: operation type, input-size band, language, model version, and whether the task modifies existing content or creates something new.
Keep the dimension set bounded. Customer identifiers, raw prompts, and arbitrary page titles create privacy and cardinality problems without necessarily improving diagnosis.
For release comparisons, examine the same workload slices and report their volumes. If a new release handles a larger share of difficult edits, a lower aggregate rate may hide an improvement within every comparable slice. Conversely, easier traffic can make a regression look like progress.
Report uncertainty and operational consequences
A percentage from twenty tasks should not carry the same confidence as a percentage from twenty thousand. Always display counts and the observation window alongside the rate.
Where a decision depends on a small difference, use uncertainty estimates appropriate to the data. Repeated tasks from one customer or one incident may be correlated, so a simple independent-trials calculation can overstate certainty.
Finally, connect the metric to a decision. Task completion can guide product reliability work. First-pass completion can expose excessive recovery. Unknown-outcome volume can reveal instrumentation or orchestration gaps. Latency can show whether successful tasks are still too slow.
The goal is not a perfect universal score. It is a small set of definitions that remain honest when a model, workload, or recovery strategy changes. A trustworthy success rate tells readers what was attempted, what was observed, and what the system still does not know.
Zahi Raifer is the founder of Orivox, focused on practical engineering for AI-assisted website building and editing.
