arXiv cs.AI· Guillaume Olivier Delplanque (TYREX), Pierre Genev\`es (TYREX), Nabil Laya\"ida (TYREX)·· 13 小时前AI 评分31
超越准确率：衡量预测模型的逻辑合规性——Rule Violation Score (RVS) 评估指标
Beyond Accuracy: Measuring Logical Compliance of Predictive Models
AI 导读
研究提出 Rule Violation Score（RVS），一种独立于预测准确率的补充评估指标，用于量化预测模型对给定逻辑规则的遵守程度。RVS 区分硬规则（严格约束）与软规则（统计规律），可对任意数据集和基于关系词汇表的预测模型计算，并能为 Horn 规则自动生成 SQL 查询完成计算。在知识图谱链接预测与关系回归三个基准上的实验显示，预测准确率相当的两个模型逻辑合规性可能差异显著。
正文 · 原文
Abstract:Machine learning models are predominantly evaluated through predictive performance metrics such as ranking quality, prediction error, or classification accuracy. While these metrics effectively quantify how closely predictions match the ground truth, they do not assess whether model outputs respect predefined logical or domain-specific constraints. In high-stakes applications, including healthcare, finance, and autonomous systems, logical consistency can be as critical as predictive accuracy, yet no standard metric captures this dimension. We introduce the Rule Violation Score (RVS), a complementary evaluation metric that quantifies the extent to which a predictive model respects a given set of logical rules, independently of predictive accuracy. RVS treats hard rules (strict constraints) and soft rules (statistical regularities) differently, can be evaluated on any dataset and on any predictive model expressed over a relational vocabulary, and can be computed using SQL queries that are automatically generated for Horn rules. Beyond evaluating models, RVS can also evaluate the logical consistency of training datasets and help identify poorly defined rules. We evaluate RVS on three benchmarks covering knowledge graph link prediction and relational regression, including rule-based, embedding-based, and neuro-symbolic predictive models. Our results demonstrate that two models achieving comparable predictive accuracy can exhibit substantially different levels of logical compliance, revealing differences in model behavior that standard metrics fail to capture.
| Subjects: | Artificial Intelligence (cs.AI); Databases (cs.DB); Neural and Evolutionary Computing (cs.NE) |
| Cite as: | arXiv:2606.20208 [cs.AI] |
| (or arXiv:2606.20208v2 [cs.AI] for this version) | |
| https://doi.org/10.48550/arXiv.2606.20208 arXiv-issued DOI via DataCite |
|
| Journal reference: | Can We Trust AI Evaluation? Robustness, Causality, and Risk in Modern AI Assessment Workshop - NeurIPS 2026, Sydney, Australia, Dec 2026, Sydney (Australia), Australia |
Submission history
From: Tyrex Equipe [view email] [via CCSD proxy]
[v1]
Thu, 18 Jun 2026 13:21:04 UTC (23 KB)
[v2]
Wed, 21 Oct 2026 17:50:41 UTC (49 KB)
来源：arXiv cs.AI · arxiv.org