
Agent simulators need a promotion contract
In short: Nubank's synthetic customers sped agent releases 4.8x; trust a simulator only when it preserves version order and earns live confirmation.
Nubank's agent simulator produced conversations that looked unlike its live traffic. In the bank's revised 5 November 2026 paper, 89.0% of production conversations had at most 50 user-message words; only 22.4% of simulated conversations did.
Yet the simulator helped cut the average development cycle from 21.2 days per version to 4.4. Its value sits in preserving the release decision even when the synthetic traffic looks different. An agent simulator is useful when it preserves the release decision. It needs a promotion contract based on version ordering, calibrated failure checks and live confirmation.
Nubank's simulator preserved the useful ordering
The Nubank-affiliated research team compared four deployed versions of its Card Delivery agent. For each version, it held out 1,000 production conversations and generated 250 trajectories with Snowglobe. Five binary evaluators checked failures such as customer-input verification, card-delivery data checks and response conciseness.
The synthetic transcripts were longer than production, but the structured simulator preserved the extremes that mattered. The production-best V4 version ranked best in 96.69% of Snowglobe resamples, and the production-worst V2 ranked worst in 100%. A naive two-model simulator reversed the signal: it ranked V4 worst and made it best in 0% of resamples.
Across the four versions, Snowglobe's evaluator scores had Pearson correlation of 0.74 and Kendall rank correlation of 0.67 with production. That is evidence of directional usefulness on this narrow set, not a universal fidelity certificate. Two middle versions still swapped places.
| Promotion signal | What Nubank reported | What it can support |
|---|---|---|
| Production-short conversations | 89.0% at 50 words or fewer | Transcript realism was incomplete |
| Snowglobe-short conversations | 22.4% at 50 words or fewer | Style similarity cannot be the main gate |
| Version-order agreement | Pearson 0.74; Kendall 0.67 | Directional screening across four versions |
| Best-version resamples | 96.69% for V4 | Confidence in one production extreme |
| Worst-version resamples | 100% for V2 | Confidence in the other extreme |
These figures describe Nubank's four Card Delivery versions, not a general benchmark for customer-agent simulators.
A promotion contract separates screening from proof
Snowglobe's product page promises hundreds of persona-driven conversations in minutes, judge-labelled datasets and regression suites. The Nubank paper gives that workflow a useful boundary: the simulator nominates candidates for live testing. It does not certify production.
The team held the scenario configuration fixed, ran the same evaluators on incumbent and candidate trajectories, and required a prespecified criterion before a candidate became eligible for an A/B test. Every new hypothesis-specific judge also had to be calibrated separately. That makes the workflow a release pipeline built with synthetic data.
My operating rule is: a simulator may reject a candidate, but it may only recommend promotion. Production outcomes still own the final decision. That asymmetry is useful because offline screening is cheap and customer exposure is expensive. A failed screen saves a risky experiment; a passed screen has only earned a better-controlled one.
The promotion sheet I would require
I did not run Snowglobe, inspect Nubank's production traces or reproduce its A/B tests. The promotion sheet below is my editorial judgement from the paper's method, not a claimed Trigent or client deployment.
For every screening round, record the incumbent and candidate identities, the fixed scenario distribution, standing checks, the one falsifiable change hypothesis, evaluator calibration evidence and the threshold that makes the candidate eligible for live traffic. Keep the production metric beside it. If offline rank improves but live self-service, satisfaction or harm moves the wrong way, the simulator loses promotion authority until it is recalibrated.
The live results show why the second half matters. Card Management beat Card Delivery by 4.90 percentage points on self-service rate and 36.69 points on transactional NPS in an A/B test. Later, after more than 16,000 simulated conversations across 29 open-weight configurations, a screened Qwen3.5 replacement improved self-service by 8.82 points and reduced p95 latency by 25%; tNPS was statistically unchanged.
Those are separate comparisons with separate controls. They confirm that screened changes produced useful outcomes. They do not prove that every offline score predicts the size of a live improvement.
Tool-boundary simulation leaves a deliberate hole
Nubank mocked stateful or side-effecting tools with schema-compatible synthetic responses. The agent still chose tools and controlled their sequence, which exposed policy and tool-call failures without touching production backends.
The boundary is explicit. This setup did not validate backend behaviour, authentication, persistent state, latency or real side effects. A record can remain perfectly consistent inside a simulated conversation while the production service rejects the request, times out or writes the wrong state.
That is why the promotion contract needs two receipts. The simulation receipt says the candidate behaved better under a controlled scenario mix. The live receipt says the integrated system improved the customer outcome without breaking another one. Mixing them into one score would turn a useful screen into false assurance.
Where simulator promotion holds and breaks
The rule holds when candidate changes can be exercised on-policy with stable tool schemas, comparable scenarios and outcome checks that distinguish incumbent from candidate behaviour. It is particularly useful when live experimentation is costly or risky, but a controlled A/B test remains possible after screening.
It breaks when the simulator's scenario mix omits the failure you care about, when a judge has not been calibrated, or when success depends on real backend state and timing. Four deployed versions are also a small basis for correlation. Treat Nubank's numbers as a strong production case study, not a transferable threshold.
What's in it for you
- You can screen more agent changes without confusing synthetic traffic with production proof.
- You get an auditable reason for why one candidate advanced and another stopped.
- You preserve live customer outcomes as the authority that can revoke the simulator's promotion rights.
Promote the simulator only when it preserves release decisions; promote the agent only when live outcomes agree.


