跳到正文
arXiv cs.AI· Vedant Navale·· 10 小时前AI 评分34

MoE-Context:让模型只看到它需要的内容

MoE-Context: Show the Model Only What It Needs

AI 导读

MoE-Context 将上下文分块,每步前用一次快速的是/否判断决定需要哪些块,其余块保留为一行摘要且不删除,并在单一 KV cache 上以掩码方式隐藏。在笔记本上用 Qwen3-1.7B 跑合成查找任务,只显示所需块把准确率从 120 题中的 93 提升到 119,掩码块与删除块效果相当。

正文 · 原文

View PDF HTML (experimental)

Abstract:AI agents read their whole history at every step. People do not work like that: we keep what the moment needs and hold the rest as a rough note we can recall. MoE-Context applies this to language models. The context is split into blocks. Before each step a fast yes or no pass decides which blocks the step needs; the rest stay as one-line notes that can be brought back, and nothing is deleted. In the version we build, one model makes this decision and then answers, over a single key-value cache where hidden blocks are masked rather than removed. On a laptop with Qwen3-1.7B and a synthetic lookup task, showing only the needed block raises accuracy from 93 to 119 of 120, and a masked block is as hidden as a deleted one. The model's own picking is the weak part. A supervised linear probe written into spare output rows lifts its accuracy from 80 to 110 of 120, but simple keyword search picks perfectly on this task, and the probe does not transfer to a second task. We report these results openly, along with two cheap usage patterns that block switching enables: undoing a failed attempt and running subagents without a new context.
Comments: 10 pages, 10 figures
Subjects: Artificial Intelligence (cs.AI); Computation and Language (cs.CL); Machine Learning (cs.LG)
Cite as: arXiv:2611.24379 [cs.AI]
  (or arXiv:2611.24379v1 [cs.AI] for this version)
  https://doi.org/10.48550/arXiv.2611.24379

arXiv-issued DOI via DataCite (pending registration)

Submission history

From: Vedant Navale [view email]
[v1] Wed, 18 Nov 2026 13:08:14 UTC (68 KB)

来源:arXiv cs.AI · arxiv.org