A success percentage needs a sentence defining success
An IIT Kharagpur probability lecture offers a practical correction for AI dashboards: define the experiment and event before comparing the percentages.
TL;DR: I would reject a success percentage until the team can finish the sentence “one trial means…” This IIT Kharagpur introduction to probability, taught by M. Chakraborty, is useful background for anyone reading AI evaluation dashboards. Its discussion of outcomes and events forces a definition before a calculation.
The lecture spends time on the language needed to say what can happen and which outcomes count together. NPTEL’s course syllabus places that foundation within a wider treatment of probability and random processes. The practical benefit is easy to miss if you are impatient to reach the formula.
In an agent system, a request, a tool call and a completed task can all become the denominator of a success rate. They answer different questions. A number alone does not reveal which one you have.
Three counts can describe the same run
Take a constructed example: 100 user requests produce 120 tool calls because some attempts are retried. Of those calls, 100 return a technically valid response. Reviewers judge 80 of the original requests successfully completed.
The tool-call success rate is about 83%. The completed-request rate is 80%. The system also generated a response to every request, which someone could report as a 100% response rate. All three calculations can be arithmetically correct. Only one describes the fraction of user requests completed to the stated standard.
I would want that distinction visible in the dashboard label. “Success” is too broad when a returned API response can still leave the user’s task unfinished.
The example also shows why retries belong in the record. If a system improves completion by spending more attempts, that can be a worthwhile trade. Hiding the extra calls would make the reliability improvement look cheaper than it was.
Write the event before observing the score
For an illustrative document-review task, I would define completion as finding the required issues, citing the relevant passages and handing unresolved ambiguity to a reviewer. Then I would state how partially completed requests are counted.
A team might choose a different event for a different purpose. Measuring whether the retrieval service answered is useful when diagnosing infrastructure. Measuring whether the final review was accepted is useful when evaluating the workflow. The problem begins when one is offered as evidence for the other.
I would keep the sampling procedure with the event definition too. A set of easy demonstrations and a sample of incoming work can produce different results even when the scoring rule is identical. An observed percentage is an estimate from that experiment, not a guarantee about every future request.
The lecture supplies foundations, not a complete methodology for evaluating agents. It does not establish independence between steps, an appropriate sample size for your task or a trading career from learning probability. Those would require additional work.
The immediate exercise is smaller: take one number from your dashboard and write its trial unit, event and sampling rule in plain language. If the team disagrees about the sentence, fix the definition before celebrating the score.
A success rate becomes useful when everyone agrees what was tried and what counted as success.
