arXiv cs.CL· Edesio Alcoba\c{c}a, Kevin Rossell, Aman Gupta, Shao Tang, Jiwoo Hong, Pabel Carrillo-Mendoza, Wanderson Concei\c{c}\~ao Ferreira, Alvaro Tedeschi, Zayd Simjee, Shreya Rajpal, Bruno Finardi Hime, Christian Sousa, Luis Moneda, Herbert Fei, Daniel Silva, Rohan Ramanath·· 12 小时前AI 评分42
部署前先模拟：Nubank 用 Snowglobe 在 1.4 亿规模下筛选客服 AI 智能体
Screen Before You Serve: Simulation for Production Customer Experience AI Agents at 140M Scale
AI 导读
Nubank 用 Snowglobe 模拟器在部署前筛选客服 AI 智能体，在 Card Delivery 及其后继 Card Management 的 4 个已部署版本上，模拟与生产的版本级二元评估分数高度相关。
正文 · 原文
Authors:Edesio Alcobaça, Kevin Rossell, Aman Gupta, Shao Tang, Jiwoo Hong, Pabel Carrillo-Mendoza, Wanderson Conceição Ferreira, Alvaro Tedeschi, Zayd Simjee, Shreya Rajpal, Bruno Finardi Hime, Christian Sousa, Luis Moneda, Herbert Fei, Daniel Silva, Rohan Ramanath
Abstract:Customer experience (CX) agents use tools and large language models to address customer requests and guide conversational interactions with an organization's products. Improving these agents, especially in regulated industries, is difficult: they must detect intent, follow complex operational policies and use tools reliably. Manual end-to-end testing offers limited coverage, while live experiments expose customers to failures that can erode trust.
We present a hypothesis-driven simulation workflow for screening candidate CX agents before deployment. Synthetic customers react to agent responses and simulated tool outputs enable multi-step agentic workflows without invoking production backends. We use the Snowglobe simulator on Nubank's Card Delivery agent and its expanded successor, Card Management - Nubank's highest-volume chat-support agent in Brazil. Across 4 deployed versions, simulated and production version-level binary evaluator scores show high correlation. Simulation-guided iteration increased transactional net promoter score (tNPS) by 36.69 points in a live A/B test. We also screened open-weight configurations in over 16,000 simulated conversations. In a subsequent live A/B test, the selected model increased self-service rate (SSR) by 8.82 percentage points to the highest level observed at Nubank, with no statistically significant change in tNPS. Simulation made broad exploration of models, reasoning settings, and prompts feasible without customer exposure, enabling production improvements that would have been impractical to pursue through live experimentation alone.
| Comments: | 17 pages, 11 figures |
| Subjects: | Artificial Intelligence (cs.AI); Computation and Language (cs.CL) |
| Cite as: | arXiv:2609.30137 [cs.AI] |
| (or arXiv:2609.30137v2 [cs.AI] for this version) | |
| https://doi.org/10.48550/arXiv.2609.30137 arXiv-issued DOI via DataCite |
Submission history
From: Aman Gupta [view email]
[v1]
Thu, 24 Sep 2026 17:07:38 UTC (2,728 KB)
[v2]
Thu, 5 Nov 2026 18:38:42 UTC (2,728 KB)
来源：arXiv cs.CL · arxiv.org