arXiv cs.CL· Sowjanya Karanam, Jayanth Bhargav·· 12 小时前AI 评分24
用蜕变测试评估情感分析模型的鲁棒性
Metamorphic Testing for Evaluating Robustness of Sentiment Analysis Models
AI 导读
一项基于批处理的轻量级蜕变测试方法被提出，用于评估情感分析等语言类 AI 系统的鲁棒性。研究对预训练分类器施加同义词替换、错字注入、语序变化和否定四种变换，并用一致性分数、翻转准确率和置信度变化三项指标衡量表现。结果显示模型在语义变化上较稳健，但对文本噪声和逻辑变换存在脆弱性。
正文 · 原文
Abstract:Machine learning systems are challenging to test using traditional software testing methods because their behavior is learned from data rather than explicitly programmed. This challenge is particularly significant in natural language processing (NLP), where small input variations can lead to unexpected model behavior. Conventional evaluation metrics, such as accuracy and F1-score, often fail to capture such inconsistencies. Metamorphic testing addresses this limitation by evaluating expected relationships between original and transformed inputs. With the increasing prevalence of generative artificial intelligence (GenAI) and language-based AI systems, the need for systematic robustness testing has become even more pronounced. Motivated by this growing need, this paper presents a lightweight, batch-based metamorphic testing approach for evaluating the robustness of such systems. In particular, we consider sentiment analysis as a representative NLP task and evaluate a pretrained classifier under four transformations: synonym replacement, typo injection, word-order variation, and negation. Using consistency score, flip accuracy, and confidence change, we identify strengths in handling semantic variations and vulnerabilities to textual noise and logical transformations. These findings highlight the value of metamorphic testing in identifying behavioral inconsistencies beyond conventional performance metrics and demonstrate its potential as a systematic framework for evaluating the robustness of language-based AI systems and guiding performance improvements before deployment.
| Comments: | 8 pages, 3 figures |
| Subjects: | Machine Learning (cs.LG); Artificial Intelligence (cs.AI); Computation and Language (cs.CL) |
| Cite as: | arXiv:2611.02248 [cs.LG] |
| (or arXiv:2611.02248v1 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2611.02248 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Sowjanya Karanam [view email]
[v1]
Sun, 1 Nov 2026 22:10:45 UTC (91 KB)
来源：arXiv cs.CL · arxiv.org