arXiv cs.MA· Mark Lovett, Denis Lim, Soroush Vosoughi·· 9 小时前AI 评分31
InflEns:用博弈论数据分配实现 LoRA 专家专业化与路由
Game-Theoretic Data Allocation for Expert Specialization and Routing
AI 导读
InflEns 提出一种混合 LoRA 专家框架,训练权重与推理路由来自固定提示词特征空间中的同一位置博弈,无需训练显式门控即可媲美已发表的学习型门控。在七个安全相关基准组成的语料上,份额最高的专家在约 90% 的提示词上取得最低的留出响应 NLL,而打乱分配后该比例约为 15%。
正文 · 原文
Abstract:Can the allocation used to train experts also predict which expert will perform best on a prompt? Such a connection would make the intended division of labor inspectable before training and allow the allocation itself to guide inference. We introduce InflEns, a mixture of LoRA experts whose training weights and inference routes come from the same location game in a fixed prompt-trait space. Players choose positions to maximize their expected shares of the prompt distribution, without using expert losses. We utilize the game's kernel-general theoretical criterion for when co-located positions become locally unstable to separation. We evaluate InflEns on a corpus assembled from seven safety-related benchmarks, which demonstrates that the highest-share expert has the lowest held-out response negative log-likelihood (NLL) on approximately 90% of prompts across four kernels under a sharpened simplex representation, compared with approximately 15% after permuting allocations across prompts. We demonstrate that the method performs comparably to published learned gates without having to train an explicit gate on the experts. Because the router is fixed, we can trace harmful behavior to a single expert and retrain only that expert, which lowers HarmBench attack success from 0.51 to 0.02 while keeping instruction following. The game also gives us several controls over how the experts specialize, including the kernel choice and its parameters, the trait-space geometry, and the number of experts. Importantly, InflEns is a general framework that can be extended to any encoder and data distribution.
| Subjects: | Machine Learning (cs.LG); Artificial Intelligence (cs.AI); Computation and Language (cs.CL); Multiagent Systems (cs.MA) |
| Cite as: | arXiv:2611.25167 [cs.LG] |
| (or arXiv:2611.25167v1 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2611.25167 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Mark Lovett [view email]
[v1]
Wed, 18 Nov 2026 20:46:49 UTC (2,844 KB)
来源:arXiv cs.MA · arxiv.org