Meta 论文介绍大规模 AI 辅助 A/B 实验的 Agent 设计与经验教训

Rohan Paul· @rohanpaul_ai · X·2026-11-08 22:01· 1 小时前AI 评分53

AI 导读

Meta 发布论文《Learnings and Pitfalls from AI-Assisted A/B Experimentation at Scale》，介绍其 ABTest Agent：LLM 只负责选择和排序实验步骤，所有数字都由经过验证的确定性工具计算，报告模板要求每张表格可追溯到代码，知识库仅收录专家审核过的文档。

正文 · 原文

New Meta paper shows that Meta’s staff can trust an AI assistant’s numbers about product tests because the AI only plans the steps and never writes a single number itself.

A/B tests need numbers people can trust, but LLMs are nondeterministic and can hallucinate. Experiments still need lots of messy manual work, like debugging setups and diagnosing odd results.

Meta's ABTest Agent lets the LLM choose and order steps, while validated tools compute every number it reports. A report template blocks output unless every table traces back to that code, and the knowledge base holds only expert-vetted docs.

Plain result fetching was 45% of requests, while 52 of 97 skills went to checking whether results can be trusted. In some sessions, the agent suggested p-hacking, such as dropping unfavorable days, when experimenters pushed for a significant result.

If you build an analytics agent, let code produce every number, and fix success metrics before the experiment starts so the agent can't help p-hack.

来源：Rohan Paul · x.com