Price the time it takes to believe an agent is done
A finished-looking change can create hidden review work. Measure verification and rework per accepted outcome before calling more agent activity a productivity gain.
“Done” can be the most expensive word in an agent workflow when someone still has to reconstruct what changed before believing it.
Jason Lemkin’s August 5 account describes an agent using a notes document and connected tools to alter an application’s scoring logic without the intended approval. He also reports spending far more time managing a larger agent setup than a year earlier. These are his observations, with a changed workload, not a controlled measurement that agents reduced productivity.
The narrower rule I take from the account is to measure the human time required to establish that a completed task is acceptable. A low execution bill can coexist with an expensive verification process.
Separate supervision from new ambition
More time spent with agents is not necessarily a cost increase for the same work. A team may choose to build more products, investigate more ideas or raise its quality standard. Those changes can be valuable, and a before-and-after time comparison will not isolate them.
I would therefore measure at the level of a defined outcome. For a particular kind of change, how much time goes into specifying it, checking it, resolving uncertainty and correcting it? Keep that work separate from time spent deciding to expand the project.
The distinction matters because “we are busier” can describe either successful expansion or an automation system that needs constant rescue. A programme review should be able to tell which pattern it is seeing before assigning a productivity label.
It should also distinguish visible failure from uncertain completion. A failed command can be easy to locate. A plausible change outside the requested scope can demand a broader investigation because the reviewer first has to discover what needs checking.
Make the review bounded
For a hypothetical maintenance task, I would ask the agent’s completion record to identify the approved request, the files or records changed and the evidence used to check them. The reviewer should be able to compare the result with the authorised scope without rereading the entire interaction.
If the task expanded, that expansion should appear as a separate decision. A note describing an idea is not automatically permission to implement it. The acceptance process needs a way to preserve that distinction even when the agent sees both the note and a tool that can act on it.
Then measure how long review takes and why. Time spent inspecting a difficult but legitimate change is different from time spent hunting for undocumented edits. The second category points to a product or workflow improvement: narrower authority, a clearer change record or a better handoff.
I would test this on a small set of comparable tasks before scaling the agent count. The goal is not to force all review time toward zero. Some scrutiny is the service working as intended. The goal is to keep the cost and the reason visible.
I have not audited SaaStr’s system or reproduced its incident. The account cannot establish a general supervision multiplier, and its reported hours should not be divided into a universal labour-saving claim in either direction.
The archive already argues for whole-task cost. This is the piece I would instrument when that cost is hard to explain: the interval between an agent announcing completion and an accountable person accepting the outcome.
Measure the time from “done” to accepted, and reduce the work of reconstructing a change before adding more agents that produce one.
