arXiv cs.CL· Qazi Mamunur Rashid, Xuan Yang, Zhengzhe Yang, Yanzhou Pan, Erin van Liemt, Darlene Neal, Kshitij Pancholi, Jamila Smith-Loud·· 13 小时前AI 评分58
NodeSyn：面向 AI 评估的社会对齐合成数据方法
NodeSynth: Socially Aligned Synthetic Data for AI Evaluation
AI 导读
论文提出 NodeSyn 方法，用微调的分类体系生成器（TaG）结合可信公开来源与敏感属性，生成反映真实社会语境和细粒度危害的查询。
正文 · 原文
Abstract:Recent advancements in generative AI facilitate large-scale synthetic data generation for model evaluation. However, without targeted approaches, these datasets often lack the sociotechnical nuance required for sensitive domains. We introduce NodeSyn, a methodology that pairs a fine-tuned taxonomy generator (TaG) with credible published sources and extracted sensitive attributes to generate queries reflecting authentic societal contexts and nuanced harms. Evaluated across ten open-weight and proprietary LLMs in Medical Advice and Self-Harm using human-validated autoraters, NodeSyn ranked first on 8 of 10 models in each domain. NodeSyn datasets induced average failure rates 3.1 times higher than generic synthetic baselines, 2.4 times higher than structured synthetic benchmarks, and 1.6 times higher than authentic human-authored queries. Ablation studies confirm that, controlling for structural confounds, our granular Level 3 taxonomic expansion significantly drives these failure rates over Level 2 across 9 of 10 models (+29.2% relative lift in Medical Advice and +24.0% in Self-Harm). Variance attribution analysis reveals that failures are driven not only by what topics are queried (taxonomic nodes and lexical specificity), but also by their sociotechnical grounding (affected demographics, occupations, geographies, and querying personas). We embed this method into a research prototype, and open-source (this https URL) the code and datasets to enable scalable, high-stakes model evaluation
| Subjects: | Machine Learning (cs.LG); Computation and Language (cs.CL) |
| Cite as: | arXiv:2605.14381 [cs.LG] |
| (or arXiv:2605.14381v3 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2605.14381 arXiv-issued DOI via DataCite |
Submission history
From: Yanzhou Pan [view email]
[v1]
Thu, 14 May 2026 05:06:50 UTC (407 KB)
[v2]
Sat, 16 May 2026 23:33:47 UTC (408 KB)
[v3]
Tue, 20 Oct 2026 21:38:55 UTC (545 KB)
来源：arXiv cs.CL · arxiv.org