跳到正文
arXiv cs.CY· Jennifer Zhang, Amanda Coston·· 23 小时前AI 评分36

研究:LLM 何时能建模正确的预测目标?高风险风险预测中的问题形式化

When Do LLMs Model the Right Target? Problem Formulation in High-Stakes Risk Prediction

AI 导读

研究评估 OpenAI 与 Anthropic 的 11 个模型在儿童福利、医疗、就业场景中生成预测模型代码时能否正确处理"决策依赖"问题,发现无提示引导时 LLM 无法可靠应对。

正文 · 原文

View PDF HTML (experimental)

Abstract:Generative AI is increasingly used to assist with data science tasks, yet fundamental questions remain about its ability to correctly formulate analytical problems. We investigate how well large language models (LLMs) can formulate prediction problems when developing models for high-stakes decision making. We focus on one particular challenge to validity: decision dependency, which arises when historical decisions affect observed outcomes. We evaluate this aspect of problem formulation in LLM-generated code and explanations across child-welfare, healthcare, and employment settings, using 11 models from OpenAI and Anthropic and seven prompt conditions. We assess whether the LLM-generated code implements a procedure appropriate for estimating the intended outcome under the study's assumptions and whether the explanation acknowledges decision dependency. We find that LLMs do not reliably address decision dependency without guidance in the prompt. When prompts specify the target, the assumption needed to estimate it, and the estimation procedure, target-aligned implementation reaches 99-100%. Among frontier models, a simple prompt description of how decisions affect outcomes appears sufficient to elicit target-aligned implementation. Some responses also acknowledge decision dependency without addressing it in their code. Our findings suggest that analysts should not rely on LLMs to correctly formulate prediction problems without explicit methodological guidance and careful review. We propose validity-centered guidelines for LLM-assisted predictive modeling, covering prompt formulation, validation of the target and estimation procedure, and identification of key assumptions. Establishing validity is a necessary early step in evaluating whether to deploy a predictive model in a high-stakes setting, but deployment requires weighing broader ethical, institutional, and practical considerations.
Subjects: Computers and Society (cs.CY); Machine Learning (cs.LG); Methodology (stat.ME)
Cite as: arXiv:2611.23638 [cs.CY]
  (or arXiv:2611.23638v1 [cs.CY] for this version)
  https://doi.org/10.48550/arXiv.2611.23638

arXiv-issued DOI via DataCite (pending registration)

Submission history

From: Jennifer Zhang [view email]
[v1] Wed, 18 Nov 2026 00:02:41 UTC (60 KB)

来源:arXiv cs.CY · arxiv.org