arXiv cs.AI· Manzanarez Gerardo Aleman, Monroy Raul, Garcia Flores Jorge, Calvo Hiram·· 4 小时前AI 评分23
GrAImes 框架研究:用 LLM 评估文学小说可行吗
Literary Language Mashup: Curating Fictions with Large Language Models
AI 导读
研究提出在 GrAImes 框架内让 LLM 充当文学小说评估者,对西班牙语人类创作与 AI 生成的微型小说、以及西班牙语和英语短篇故事进行打分,并与 5 位文学博士教授和 16 位文学爱好者的评判对比。结果显示 LLM 与人类评委的一致性不足,Krippendorff's alpha 信度系数低于 0.66,无法替代人类评委,但可作为初步判断文本编辑质量的辅助工具。
正文 · 原文
Abstract:In this paper, we propose an alternative approach to human evaluation of literary fictions by employing LLMs themselves as evaluators within the GrAImes framework. We apply this methodology to assess human-written and AI-generated microfictions in Spanish, to five PhD professors in literature and sixteen literary enthusiasts, and to short stories in both Spanish and English. By comparing the evaluations performed by LLMs with those of human judges, we examine the degree of alignment and divergence between both perspectives, thereby assessing the feasibility of LLMs as auxiliary literary evaluators. Our analysis focuses on the alignment of responses from LLMs with those of human evaluators, providing insights into the potential of LLMs in literary assessment. The conducted experiments reveal that while LLMs cannot be regarded as substitutes for human judges in the evaluation of literary microfictions and short stories, with a Krippendorff's alpha reliability coefficient less than 0.66, they can serve as a valuable tool that offers an initial perspective on the editorial quality of the texts in question. Overall, this study contributes to the ongoing discourse on the role of artificial intelligence in literature, underlining both its methodological constraints and its potential as a complementary resource for literary evaluation.
| Subjects: | Computation and Language (cs.CL); Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2611.21649 [cs.CL] |
| (or arXiv:2611.21649v1 [cs.CL] for this version) | |
| https://doi.org/10.48550/arXiv.2611.21649 arXiv-issued DOI via DataCite (pending registration) |
|
| Journal reference: | Mathematics 2025, 1, 0 |
| Related DOI: | https://doi.org/10.3390/math1010000
DOI(s) linking to related resources |
Submission history
From: Gerardo Aleman Manzanarez [view email]
[v1]
Thu, 15 Oct 2026 01:43:14 UTC (2,823 KB)
来源:arXiv cs.AI · arxiv.org