arXiv cs.CL· Xiaozhe Ji, Renfen Hu, Hai Hu·· 11 小时前AI 评分29
语言模型掌握多少汉字语音知识？PinyinBench 基准揭示声调信息最难获取
Phonological Knowledge of Chinese Characters in Language Models
AI 导读
研究者提出 PinyinBench 基准，覆盖同音字、声母韵母声调等语音特征及多音字，评测 17 个纯文本与多模态语言模型。前沿模型 DS-v4-flash 在同音字判断中得分 90%+，但两字仅声调不同时准确率降至 72%。探测显示声调可从嵌入向量中高度解码，而分段信息随层深显著提升；激活引导对分段表征有效，对声调影响微弱。
正文 · 原文
Abstract:How much phonological knowledge do language models (LMs) possess about Chinese characters, and how is this knowledge represented and used? We introduce PinyinBench, a controlled benchmark covering homophones, phonological features (initials, finals, and tones), and polyphonic characters. We investigate 17 text-only and multimodal LMs through three complementary approaches: behavioral evaluation, layer-wise probing, and activation steering. Behaviorally, models demonstrate substantial knowledge of initials and finals, with a frontier model (DS-v4-flash) scoring 90%+ in homophone judgment, when two characters share no phonological features. However, its accuracy drops sharply to 72% when two characters differ only in lexical tone. Probing reveals that tone is highly decodable from embeddings and improves modestly across layers, whereas segmental information starts less decodable and improves substantially with depth. Activation steering shows that segmental representations can systematically influence generation, while tone exhibits much weaker effects. Furthermore, audio-bearing training improves phonological decodability, including tone, without corresponding improvements in steering effectiveness. Together, these findings reveal distinct representational and functional profiles for segmental and tonal information, with tonal information much harder to access and manipulate, demonstrating the necessity for combined evaluation, probing, and causal intervention for understanding linguistic knowledge in LMs.
| Comments: | 17 pages, 11 figures, 10 tables |
| Subjects: | Computation and Language (cs.CL) |
| Cite as: | arXiv:2611.05803 [cs.CL] |
| (or arXiv:2611.05803v1 [cs.CL] for this version) | |
| https://doi.org/10.48550/arXiv.2611.05803 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Xiaozhe Ji [view email]
[v1]
Wed, 4 Nov 2026 01:22:05 UTC (1,549 KB)
来源：arXiv cs.CL · arxiv.org