Redwood Research 提出 NLS depth：量化模型不可见串行推理的新指标

目前，思维链（CoT）是监督 AI 模型的重要工具。然而，一些架构转变可能会显著降低 CoT 的可监控性。我们最近提议，AI 公司应透明地分享有关其架构可能允许潜在推理和通信程度的信息。为支持这一提议，本文档将一项度量操作化，作为模型可执行的未言明串行认知量的代理指标。我们的度量是“不透明串行深度”这一概念的具体实例，该概念最初由 GDM 近期的一篇论文（Brown-Cohen et al, 2026）提出。

为了测量计算的不透明串行深度，Brown-Cohen 等人提出测量计算图中不经过某种“可解释瓶颈”的最长路径。核心在于，如果将 CoT token 视为“可解释”的，而将 transformer 隐藏状态视为“不可解释”的，那么标准 transformer 的不透明串行深度与层数成正比。^[1]

本文档的主要贡献在于提出了一个关于何为“可解释瓶颈”的具体标准。粗略地说，我们希望将那些输出文本（而非潜在状态）且从预训练先验初始化的节点视为“可解释瓶颈”。为使这一概念更精确，我们定义了计算图中自然语言扎根节点（NL-rooted 节点）的概念。^[2]要使模型计算图中的一组节点被视为 NL-rooted，我们要求：

• 自然语言初始化：模型已从预训练先验初始化，以在这些节点产生“自然语言”输出。（我们对自然语言的定义非常宽松，包括“代码”、“json”和“图像块”等对象，并允许从 CoT 混乱的模型进行蒸馏。）

• 不扩展输出空间：这些节点的唯一输出是其含义直接从该预训练先验初始化的 token。（排除例如抽象 CoT、COCONUT、全带宽 transformer和token 叠加。）

• 不修改模型以将这些 token 解释为不同类型的数据：不应直接修改模型，以与预训练先验完全脱节的方式使用这些 token。（本条款不应排除自然训练方法。它主要旨在排除上述要点中的技术漏洞，例如将潜在推理模型通过将其激活编码和解码为自然语言 token，转变为表面上用自然语言推理的模型。）^[3]

我们将在下文详细阐述这些标准。重要的是，所有常规形式的训练后处理都可以使用，而不会阻止节点成为 NL 根节点，包括提示、蒸馏和强化学习。（即使采用激进的长度惩罚或 CoT 监督，即使这些形式的训练后处理在实践中使 CoT 变得不可读。）

基于 NL 根节点的这一定义，我们将相应的不透明串行深度概念称为“自然语言根节点”分离深度（NLS 深度）。除了定义这一术语外，我们还论证它追踪了令人担忧的架构变化，讨论了如何将 NLS 深度应用于若干示例架构（见附录 A），并讨论了一些不透明串行深度的替代操作化方案（见附录 B）。

追踪 AI 系统的 NLS 深度并不能替代追踪可监控性或无 CoT 能力；它是一个重要的补充。NLS 深度具有其他这些测量方法所不具备的一些优势。具体而言：

• NLS 深度是一个相对清晰且易于量化的概念，^[4]而如何量化或测量可监控性尚不明确（例如，应在哪些任务上测量，应使用哪些引导策略）。

• NLS 深度可以在训练开始前就仅根据 AI 系统的架构、训练配方和预期部署时配置来判定。这使得它更容易用于决定训练哪些模型。

为实现这些特性，NLS 深度对某些训练后处理选择不敏感，而这些选择实际上可能导致模型拥有显著更多或更少可解释的 CoT。然而，我们认为 NLS 深度是模型能够执行的不透明串行认知量的一个良好（尽管不完美）代理指标^[5]，我们认为这与可监控性高度相关。NLS 深度能够很好地将当前自然语言 CoT 范式中的架构（即使经过大规模扩展）与特别令人担忧的架构（如全带宽 transformer的放大版本，其可以在整个上下文窗口中对不透明状态进行递归）区分开来。总体而言，我们认为追踪 AI 系统的 NLS 深度将非常有价值。

自然语言根节点的定义

在定义 NLS 深度之前，我们必须定义计算图中的“自然语言根”（NL 根）节点是什么——这些节点将在我们对不透明串行深度的定义中充当“可解释瓶颈”。

Brown-Cohen 等人提出了两个不同的候选方向来定义什么算作可解释信息瓶颈。我们的 NL 根节点概念与第二个方向（在其附录 A.2中提出）相似，并且具有相似的动机：

直观地说，LLM 思维链之所以可解释，关键原因在于我们训练 LLM 预测自然语言，从而为 LLM 提供了一个“自然语言先验”。因此，模型对思维链中 token 的语义应当模仿人类会赋予这些 token 的语义，因为这正是预测人类生成文本时有用的东西。

受这一基本论点的启发，我们提出两条标准，二者共同表明一个中间节点应被视为可解释的：

1. 自然语言先验：中间表示被直接优化，要么 (1) 模仿人类书写的文本（例如通过预训练或监督微调），要么 (2) 对人类看起来良好（例如 RLHF（Ziegler et al., 2019））。

2. 对其他激励的有限优化：除自然语言先验之外，存在相对较弱的训练激励（理想情况下没有）。直接与自然语言先验相悖的激励比与它正交的激励更糟。

我们选取第二条标准的一个具体且相当宽松的版本。我们的意图是让这一度量追踪当前 CoT 范式与尤其令人担忧的架构变化之间的差异，而不是试图划出一条更难定义的界线，来说明当前范式中何种类型和程度的 RL 应使推理从算作不可解释翻转为可解释。

为使一组节点被算作 NL 有根，我们要求：

• 自然语言初始化。

• 不扩展输出空间。

• 不修改模型以将这些 token 解释为不同类型的数据。

我们还强调一个由这些性质所隐含、但重要到足以单独陈述的特定属性：NL 有根节点中的 token 不应被视为一个会被反向传播穿过的连续中间状态。

我们现在将逐一展开这四条属性。

自然语言初始化：模型应已从预训练先验初始化，以模仿人类可理解的媒介，其中“NL 有根”节点最初表示来自该先验的样本。

“人类可理解的媒介”不仅包括文本，还包括代码、json 文件、图像或音频（不过请注意，多模态 CoT 多少更为棘手；见下文的讨论）。这诚然使“自然语言有根”成为一个略显不幸的术语。

“人类可理解的媒介”还被推定包括来自 NL 有根节点自身的所有输出，以便例如从其他模型蒸馏。^[6]（请注意，这意味着在某些情况下，如果模型 CoT 变得非常难以辨认，“人类可理解的媒介”最终可能变得难以理解。）如果实践了关于 NLS 深度的透明度，我们建议 AI 公司标明它们是否以及如何在蒸馏流程中使用高 NLS 深度模型，因为低 NLS 深度模型在从高 NLS 深度模型蒸馏时有可能继承某些危险。^[7]

不扩展输出空间：节点仅表示输出空间中的 token，其中几乎所有 token 都已从预训练先验中初始化，^[8]或者已单独确定具有人类可理解的含义，如 <EOS>、<UNK>、<think>、<PAD>。这旨在排除诸如 COCONUT、抽象 CoT、全带宽 transformer 以及 token 叠加 等技术。所有这些技术都将通过扩展输出空间来实现。^[9]

不修改模型以将 token 解释为不同类型的数据：我们不仅要求“以自然语言为根基的节点”继续在相同的输出空间上运行，还规定该输出空间仍应大致按照与预训练所用模型相同格式的自然语言来解释（对于普通 transformer 而言，即“续写一段文本序列”）。

也就是说，如果模型架构已被更改，以至于人们应当预期该节点的输出被解释为与它所训练模仿的人类可理解媒介完全不同类型的数据，那么该节点就不算以自然语言为根基。（RL 或提示蒸馏等常规训练方法可能会改变模型使用其输出的方式，但不包含在此定义中，因为它们不会可预测地导致 token 被用作某种其他底层数据类型的代理编码。^[10] 我们在此关注的是直接改变架构，或直接训练以使 token 成为某种其他类型数据代理的方法。）

以下是一些旨在明确排除的情况示例：

1. 优化以自然语言为根基的节点的输出，以模仿（或以其他方式非常强烈地优化模型以编码有关）AI 模型不透明状态的信息。（我们可以将其操作化为不以自然语言为根基的状态。）这将包括经过强烈优化以最优方式编码激活信息的 NLA（自然语言自编码器）。

2. 取多个 token 样本，将嵌入相加，并将组合结果反馈给模型。这将被视为改变了数据类型，因为多个不同 token 嵌入之和是模型在预训练中从未遇到过的一种输入类型，并且它在自然语言中没有明确的对应物。^[11]因此，其输出将不算作以自然语言为根基。^[12]

不将 token 视为可反向传播以训练模型权重的连续中间状态：对于当前的 CoT 模型，每次前向传播都以不可微的采样步骤结束。因此，在进行 RL 或 SFT 时，梯度绝不会从一次传播的输入层传递到上一次传播的输出层。

我们要求同样的性质（在训练中，来自后续节点的梯度不会通过该节点反向传播来更新模型权重），才能将某个东西算作 NL 根节点。我们没有明确规定模型如何采样 token，但要求采样操作本身同样被视为不可微（直接从 token 概率计算损失的算法，如策略梯度，是可以的）。

鉴于上述标准，这一标准可能是多余的。^[13] 然而，我们仍然认为这一标准值得单独提及，因为通过表示进行反向传播似乎是一种特别可能使它们变得难以监控的方式。担忧在于，通过 NL 根节点进行反向传播会导致节点表示发生变化，而这些变化不对应任何人类可理解的语义。这比 RL（或例如进化算法）的情况要严重得多，在 RL 中，搜索主要发生在源自预训练的分布内。

同样的限制也适用于其他不完全是反向传播、但也将节点视为连续并搜索对其的微小局部变化以优化其对其前一层的有用性的算法。此外，通过计算图中 紧邻 NL 根节点之前的节点进行反向传播，可能导致它不再是 NL 根节点。对于实际的 NLS 深度计算，这应该很少有关系，但可能对蒸馏有影响。详见脚注。^[14]

关于如何判定某物是否算作自然语言根节点的示例，见 附录 A。

NLS 深度的定义

我们现在基于自然语言根节点的定义来定义主要关注的度量，即“自然语言根节点”分隔深度（NLS 深度）。

NLS 深度：如果某个计算所实现的函数可以由一个可接受电路表示，且其中每条不透明路径的长度至多为 D，则该计算的 NLS 深度至多为 D。

该定义中涉及三个关键概念：

• 可接受电路：一种电路，其门是从两个 32 位值到单个 32 位值的任意函数，最多包含 10^26 个门。（特别是选择“32”位，以及特别是选择“10^26”个门，都可以用其他合理的选择替代，并且对定义基本没有实质性影响。^[15]）注意，神经网络（或神经网络系统）可以很容易地转换为可接受电路。^[16]

• 不透明路径：电路中任何不涉及任何被标记为可解释瓶颈（即自然语言根节点）的电路节点的路径（边从输入指向输出）。

• 可以用以下方式表示：最严格的解读会要求电路计算与神经网络完全相同的函数。我们稍微放宽了这一点：电路可以在不实质性改变模型输出的方面有所不同，例如重排归约顺序、改变浮点舍入方式，或移除量化。它不能在实质性改变行为的方面有所不同。例如，一个仅通过蒸馏等方式匹配原始模型基准性能的更浅层模型，不算作对它的表示。^[17]

这与Brown-Cohen等人给出的定义非常相似；唯一的区别在于我们对可接受电路的定义方式。^[18]

这个定义在技术上可以应用于任何计算，但在实践中我们希望用它来衡量AI系统的NLS深度。直观地说，“AI系统”是某种被部署来解决问题的机器学习系统（例如，一道数学题）。AI系统的一个核心例子是在一个上下文长度上针对提示运行的LLM。其他一些算作AI系统的对象包括：带压缩运行的智能体、存储记忆的智能体、在一次推理展开过程中执行多次反向传播的智能体（例如TTT中的情况），以及通信智能体集群。一个我们不想视为AI系统的计算例子是“训练AI模型的整个过程”。当LLM通过工具调用与人类或计算机交互时，我们也不认为该人类或计算机是AI系统的一部分。

以下是对上述一些AI系统的NLS深度的一些直观理解：

• 对于用自然语言CoT进行推理的模型，整个推理展开的NLS深度与单次前向传播（在最大上下文长度下进行）的NLS深度相同。即使推理展开涉及将上下文压缩为文本摘要、在上下文中存储文本记忆，甚至涉及与其他智能体进行文本通信，这一点也成立。

• 对于进行潜在推理的模型（如Meta的COCONUT），推理展开的NLS深度是潜在推理步数乘以单次前向传播的NLS深度。类似地，对于循环Transformer，NLS深度是循环迭代的深度乘以循环次数。在每种情况下，即使token与潜在状态并行生成，这一点也成立，因为仍然存在通过不透明状态的长路径。

• 如果模型存储并稍后读取不透明记忆，那么最长的不透明记忆链将决定NLS深度。

• 类似地，如果你有一个使用不透明状态进行通信的智能体集群，那么最长的不透明状态链将决定NLS深度。

给定 AI 系统的 NLS 深度会因不同的部署配置和输入而变化。当前 LLM 在更长的上下文长度下深度略高（深度随序列长度的缩放通常相当缓慢）。^[19]更重要的是，某些架构可能允许可变长度的前向传播。我们预计，实践中使用的最大 NLS 深度是最重要的追踪指标（但 NLS 深度的粗略分布，以及最大易于实现且性能良好的 NLS 深度，也可能提供有用信息）。

NLS 深度是模型相当自然的一个属性，而且计算模型深度的良好上界相当容易——我们在附录 D中给出了 Gemma3-1B 的这一计算示例。特别地，给定本文档和某个架构的细节，现有消费级 AI 模型能够计算出我们认为相当接近最优的界。标准 transformer（非稀疏注意力）的 NLS 深度大致为层数 · (4⌈log₂ hidden_dim⌉ + ⌈log₂ sequence_length⌉ + 17)。^{[20] [21]} NLS 深度是比层数更好的报告指标，因为“层”是一个非正式的定义，对于新架构，可能很难确定什么算作一个“层”。幸运的是，电路深度是一个数学上严谨的定义，它本质上捕捉了同样的东西。事实上，如图 1 所示，在开源模型中，直观的层数与电路深度高度相关。

图 1：在开源模型中，NLS 深度与层数之间存在强相关性。

注意：使用 Gated DeltaNets 的 Kimi-K3 和 DeepSeek-V4-Pro 等模型，其每层 NLS 深度显著高于图 1 中列出的其他更标准的 transformer。我们认为，使 Kimi-K3 深度更高的架构变化，从可监控性角度看确实有些令人担忧。另一方面，我们认为 DeepSeek-V4-Pro 的高深度是由于我们定义中的缺陷，因为它之所以大，只是因为计算高度串行但 FLOP 非常低。更多细节见附录 G。

在附录 A 中，我们通过列举会或不会显著增加 NLS 深度的架构变化示例，为 NLS 深度提供了更多直观理解。

NLS 深度能追踪令人担忧的架构变化

NLS 深度似乎值得追踪的一个核心原因是，处于历史 transformer 范式中的模型，其 NLS 深度会显著低于我们认为对可监控性尤其令人担忧的架构，例如潜在推理架构，这类架构能够不透明地执行大量计算。

即使对历史性的 transformer 架构进行多个数量级的扩展，也不会大幅增加深度。关于开源模型中深度扩展速度的历史趋势，请参见图 2。关于我们预期 NLS 深度在标准 CoT 范式中扩展速度的更多分析，包括基于活跃参数数量的历史扩展和基于缩放定律的分析，请参见 附录 E。

图 2：近年来开源模型未出现显著的深度扩展。

AI 开发者并未扩展不透明的串行深度，而是通过 推理时扩展 来增加计算深度，即利用更多的自然语言推理。没有任何开源 CoT 模型的 NLS 深度超过 17,000。当然，我们不知道闭源模型的深度，但如果其中任何一个的 NLS 深度超过这个数值的两倍，我们会感到惊讶。

然而，如果公司转向 潜在推理架构，NLS 深度将迅速大幅增加，如图 3 所示。

图 3：即使是一些相当小型的（假设性）潜在推理架构，其 NLS 深度也会远大于最大的开源 CoT 模型。（注意对数刻度。）

一种会导致 NLS 深度显著但不太极端增加的架构变更是循环 transformer（Geiping et al., 2025）。这些与当前范式区分度较低，因为循环 transformer 的深度可能与普通 CoT 模型相似（例如，如果将一个 120 层模型替换为两个 60 层的副本），但也可以扩展到更深的程度。对于这些模型，我们认为 NLS 深度很可能是一个不错的关注程度量化代理指标（即使通过循环实现 2 倍的深度扩展，其深度也相当于 CoT 范式内多年的扩展）。

注意：OpenAI 的最新模型 Astra 的 CoT 可监控性大幅低于其之前的模型。原因尚不清楚，但公开证据与架构变更导致更高深度是重要促成因素这一解释一致。^[22]

NLS 深度未能追踪的一个令人担忧的架构变更是：从以自然语言相互通信的深度循环模型，转向以不透明状态通信的深度循环模型。关于捕捉这一点的另一种不透明能力代理指标，请参见 附录 F。

结论

在本文档中，我们定义了 NLS 深度——不透明串行深度的一种操作化。直观地说，如果一个模型是处理文本而非不透明状态的浅层 transformer，那么它的 NLS 深度就较低。我们认为，转向 NLS 深度大幅更高的架构，很可能会导致模型具有高得多的不透明串行能力，从而更难被监督。我们认为，AI 公司追踪并公开其模型的 NLS 深度是有价值的。

附录 A：将 NLS 深度应用于可能的架构

我们现在通过讨论 NLS 深度会如何因架构或模型训练过程的各种变化而改变，来阐明我们的操作化定义。具体来说，我们将讨论一些会显著增加 NLS 深度的变化，以及一些不会显著增加 NLS 深度的变化。

不会增加 NLS 深度的例子

在训练过程中自然变得混乱的 CoT。如果 CoT 在训练过程中自然变得难以理解，这不会增加 NLS 深度，因为该 CoT 仍会被视为以自然语言为根基。这是有意为之：我们希望我们的定义能够在训练之前就得到判定。保持 CoT 清晰可读且信息丰富很重要，但这更难操作化和衡量。我们还怀疑，如果模型继续保持较低的 NLS 深度，那么找到技术方案来解决 RL 导致 CoT 保持清晰可读且忠实的问题将更为可行。

在 CoT 上训练。对 CoT 进行合理的训练并不会阻止其“以自然语言为根基”，因此不会增加架构的 NLS 深度。例如，仅仅因为训练惩罚了进行“语言切换”或以其他方式变得混乱的 CoT，并不会增加模型的 NLS 深度。尽管此类训练保持了较低的 NLS 深度，但我们建议 AI 公司在基于 CoT 进行训练时谨慎行事（并考虑不这样做），因为这可能对 CoT 的可监控性产生不利影响。恶劣形式的训练，例如故意使模型以与预训练先验完全脱节的方式解释其 token（例如，通过模仿一个被训练为使用完全不可解释的循环进行推理的 AI 的激活），将违反“不得修改模型以将 token 解释为不同类型的数据”的要求，从而使这些 token 不再以自然语言为根基。

将多个推理步骤折叠进一次前向传播。一些工作旨在通过训练模型在单次前向传播中计算思维链的多个步骤，来缩短 LLM 推理的长度。这在理念上与 COCONUT 有相似之处，因此我们将其列在此处。与 COCONUT 不同，这种架构选择不会增加 NLS 深度。这是因为以这种方式压缩推理不会增加模型前向传播的深度，不会向输出空间引入任何新内容，也不会使模型将这些 token 视为不同类型的数据。

基于编辑的序列生成。自回归 CoT 的一种被提出的替代方案是允许模型对其推理轨迹进行局部编辑（即插入和删除，而非仅追加）。此类架构的早期实验包括Levenshtein Transformer和Insertion Transformer。对于此类模型，每个推理步骤仍然是一个低深度的函数，将自然语言 token 映射为自然语言 token，并且模型对这些 token 的处理方式与预训练期间相同，因此这种架构选择不会增加 NLS 深度。

在线训练。 对定义的朴素解读会给每个模型赋予极大的 NLS 深度，因为训练会通过模型进行反向传播：通过模型进行反向传播会以自然语言之外的形式将信息从输出传回较早的层。显然，仅凭这一点不应使一个架构被视为具有高 NLS 深度。我们不认为对模型进行的反向传播是 AI 系统本身的一部分；相反，这是公司在其模型上运行的一些外部优化过程。一家公司可以让每一次推理生成都涉及大量反向传播，有效地用作在前向传递之间传递信息的方式，但只要没有发生类似的事情，训练（包括部署中的在线训练）就不会增加 NLS 深度。

AI 模型修改 AI 权重。反向传播是 AI 输出最终以不透明方式修改模型权重的一种方式。如上所述，这不会增加 NLS 深度（只要反向传播不被用作 AI 系统推理时推理的一部分，正如我们稍后将讨论的那样）。我们希望将类似的标准应用于 AI 修改自身权重（或其他 AI 权重）的任何其他反馈回路。例如，未来模型将越来越多地被用于编写训练未来模型所用的代码、生成用于训练的合成数据，甚至直接修改自身权重（例如通过找到好的引导向量）。我们认为这类操作没有问题，只要其中仍然涉及一个以自然语言为根源的瓶颈（例如，AI 编写代码，然后该代码被用于找到引导向量，而不是让架构直接将最终层激活的某一块输送出去以产生引导向量）。只有极其恶劣的形式才会增加 NLS 深度，例如训练模型在每次推理生成过程中多次更新自身权重。

仅以 token 为瓶颈的离散语言扩散。 进行语言扩散有多种不同方式，其中一些会导致 NLS 深度大幅增加，而另一些则不会。在像 LLaDA 和 Dream 这样的扩散模型中，每个去噪步骤的输入和输出都是显式的 token 序列（可能还有一些被掩码的 token），除此之外没有别的。这些 token 和扩散操作基于人类可理解的先验进行初始化，并且没有以明显会导致失去这种对应关系的方式进行显式训练，因此会被算作以自然语言为根源。因此，与其他前沿规模的模型相比，仅以 token 为瓶颈的前沿规模离散扩散模型不会具有高 NLS 深度。然而，连续文本扩散模型和像 DiffusionGemma 这样的潜在扩散模型确实具有高 NLS 深度，如下所述。

（我们怀疑，文本扩散模型产生的推理会比真正的自回归 CoT 可读性差得多。推理以更加全局化的方式发生，中间状态很可能以某种方式变得混乱，从而难以追踪正在发生的事情。我们预计，如果语言扩散模型被大规模部署并受到大量 RL 压力，这种差距会变得更加明显。因此，采用另一种定义、将离散语言扩散也归类为具有高不透明串行深度，可能是合理的。但 NLS 深度并不这样做。）

适度增加 NLS 深度的例子

Mamba。在推理时，Mamba 模型通常以循环方式评估——也就是说，不只是生成 token，一些信息会通过一个不透明的隐藏状态传递。然而，存在一种数学上等价的并行评估方式，其中下一个 token 确实可以仅根据先前的 token 生成。这需要更长时间（存在对数深度开销），但不是线性更长，而真正的循环模型会是线性更长。因此，Mamba 模型的 NLS 深度仅比可比的 transformer 模型适度更高，即使它在实践中以循环模式运行。

DeltaNets。就像 Mamba 一样，DeltaNets 通常以循环方式评估，但每个 token 也可以通过一个相当低深度的计算来计算，并带有 polylogarithmic 深度开销。^[23]因此，像 Kimi-K3 这样的 DeltaNet 架构相比标准 transformer 有适度但并非巨大的深度增加。

循环 transformer。循环 transformer在生成 token 之间会重复其前向传递若干次。如果在每个 token 之间将整个前向传递重复 k 次，这会将 NLS 深度相比一次前向传递增加 k 倍。取决于 k，这可能是 NLS 深度的适度或非常大的增加。

请注意，同一篇论文还提出了一种设计，其中 KV 缓存会在不同深度的层之间共享。这种变体会大幅增加 NLS 深度。扩展将与 token 数量成线性关系。更多内容，见 MBaert (2026)。

大幅增加 NLS 深度的例子

循环 / 软 token。放大版的 COCONUT 是具有高不透明串行深度的架构的典型例子。COCONUT 不是用自然语言 token 进行推理，而是通过将其整个隐藏状态反馈到其第一层来进行推理。隐藏状态本身并非以自然语言为根源，因为它们既利用了更大的输出空间又通过反向传播，因此每个顺序推理步骤都会增加 NLS 深度。如果 COCONUT 执行 1000 步推理，其 NLS 深度将比单次前向传递的深度大 1000 倍。

抽象 CoT。一些近期工作采用 CoT 模型，并训练它利用扩展的 token 集合，而不鼓励与人类语言有任何特定关系。与 COCONUT 不同，这些仍然是离散 token——然而，它们利用了扩展的输出空间，因此其 NLS 深度等于一次前向传播的深度乘以推理步数。

叠加。一种方法是通过创建一个由 token 嵌入的概率加权混合表示的概念 token来表达 token 上的完整概率分布，并将其馈入下一个模型。（Zhang et al., 2025；Zhuang et al., 2025；Wu et al., 2025）。这利用了扩展的输出空间（要么传递概率分布本身，要么传递预训练期间从未出现过的 token 嵌入），因此 NLS 深度将等于一次前向传播的深度乘以推理步数。即使你只是采样 token 然后将它们平均在一起，情况也是如此，因为这种用法会将 token 解释为与预训练期间不同类型的数据。（在预训练期间，token 嵌入不会那样被平均在一起。）

不透明记忆库。目前，一些 AI 模型形成基于文本的记忆，保留部分上下文以影响未来的交互。未来，模型可能会转而形成不透明记忆——例如，存储特定的激活值并将其作为未来生成的输入反馈回去。这种不透明循环在比 COCONUT 更长的时间尺度上运作，因此相对不那么令人担忧，但仍会增加 NLS 深度（而且确实似乎令人担忧）。如果这被认为不可取，我们在此处讨论了一种替代定义，该定义允许某些形式的不透明记忆而不增加不透明串行深度。

连续或潜在增强的语言扩散。虽然仅以 token 为瓶颈的离散扩散根据我们的定义不会显著增加深度，但大多数近期的语言扩散模型并不遵循这种形式。相反，它们要么是连续扩散模型，如Plaid，要么是潜在增强扩散模型，如CADD和DiffusionGemma。在前一种情况下，中间状态是文本嵌入的连续加噪版本，而非实际文本。在后一种情况下，文本扩散过程伴随着另一个在连续潜在变量上的并行过程。这两者都使用了未经预训练以与人类可理解媒介具有明确对应关系的状态，因此这些推理步骤会显著增加 NLS 深度。

（GDM 有一篇近期论文（Engels 等，2026）主张，他们通常可以通过机制可解释性理解这些潜在表示所编码的内容。这类工作是有价值的，无论对于扩散语言模型，还是对于更一般地解释模型中的隐藏状态而言都是如此。然而，我们认为这种白盒可解释性不太可能完全替代 CoT，并预测随着模型规模增大，其效果会变得更差。）

不透明模型 + NLA。 我们现在描述一种架构，它被设计成看起来具有较低的 NLS 深度，但由于与经典不透明循环架构类似的原因，仍然难以被监控。正如我们将解释的，这种尝试实际上被归类为高 NLS 深度。假设一个 AI 按如下方式产生：从一个预训练模型开始，训练一个NLA将最后一层激活转换为大块文本。通过让每一步的最后一层激活先由 NLA 编码，再由 NLA 解码，然后传入下一步，将这个模型与自身组合多次。对这个新模型应用基于结果的 RL。

有人可能会论证说，这里的中间步骤是 NL 根源的。确实，“NLA 解码、步骤、NLA 编码”的组合是一个低深度电路，它接收自然语言并输出自然语言。而且人们可能会试图论证，NLA 是通过对模型可能在想什么做出听起来合理的猜测来初始化的，而这对应于在人类可理解的先验中进行初始化。然而，如果 NLA 输出的文本块足够大，以至于获得（接近）完美的重建损失，那么这个系统在功能上就等价于不透明循环！人们甚至可以想象说，作为一种“性能优化”，NLA 编码/解码步骤在实践中会被移除。因此，似乎有必要阐明为什么这些中间状态不应算作 NL 根源的。

主要的反对意见是：这种结构似乎是在训练模型将这些 token 解释为一种不同类型的数据。这些 token 被直接优化为有助于预测模型的不透明状态（通过重建损失），这意味着它们很可能只会成为这种数据的一种任意表示，而不是表示其初始训练中的语义。

测试时训练（TTT）。 一些模型架构在单次推理 rollout 过程中会多次执行反向传播。默认情况下，这些反向传播步骤会被计入计算的 NLS 深度——反向传播是一种在步骤之间将不透明状态存储在权重中的方式。这与发生在推理 rollout 之间的在线训练形成对比（即一个不属于推理过程的单独外层循环），后者不会增加模型的 NLS 深度（这是因为在线训练看起来与模型的推理是相当分离的过程，所以我们不把它算作任何你应当追踪其 NLS 深度的“AI 系统”的一部分）。

我们对这种分类相当满意，尽管我们认为这里仍有讨论空间。以下是我们认为值得将 TTT 算作大幅增加 NLS 深度的原因：

• 在推理过程中修改权重的一些方式显然相当令人担忧：例如，如果你总是把最后一层的激活值写到一个在计算下一个 token 时第一层计算可以访问的地方，这就允许了任意深度的不透明递归。

• 原则上，应该可以写出一种模型权重配置，使得反向传播确实能够以可读的形式将最后一层激活值的所有信息传回第一层。

• 先验地看，很难排除某种特定的 TTT 方案最终会有效地做出类似的事情。

尽管如此，TTT 方案通常在我们看来远不如例如 COCONUT 那样令人担忧，而且其递归机制也相当不同。因此我们认为，合理的不透明串行深度替代定义可能会对许多 TTT 方案区别对待。（不过请参见下文“元学习测试时训练”一节，了解我们认为更令人担忧的方案。）TTT 的最简单形式，“动态评估”，除了在读取上下文时对下一 token 预测执行 SGD 之外不做其他事情。如果只在教师强制训练时（即模型读取由人类或其他模型生成的内容时）进行，这当然不会增加 NLS 深度。如果在推理或 RL 期间进行，按照我们目前的定义，它会迅速使 NLS 深度爆炸，但改变这一点也许是可以的。一个原因是至少存在梯度屏障——即可以通过更新权重将一次前向传播的最后一层的不透明信息传递到下一次前向传播的第一层，但该过程本身并不进行反向传播，因此如果将这些权重更新视为“NL 有根的”，那么该方案就是被允许的。按照我们的操作化方式，权重更新绝对不是“NL 有根的”，但人们可以想象尝试放宽定义——例如声称反向传播是与 token 预测相关的自然过程，而模型不太可能学会利用这一过程进行推理——或者干脆去掉这一要求（关于不透明串行深度的一些移除该要求的替代操作化方式，请参见附录）。

元学习测试时训练。一些现代版本的测试时训练，如果扩大规模并优化，似乎最终可能大致与一种不透明循环的版本一样危险，该版本具有有限数量的循环迭代，且仅定期应用。（除非测试时训练更新极其频繁，否则这比完整版的不透明循环要轻得多。）例如，Tandon 等人的 TTT-E2E 使用了一种“元学习”过程，其中外层循环优化权重，使测试时训练性能更高。即使有人修改我们的定义，使得 动态评估（如上文所述）不会增加不透明串行深度，我们认为这种修改不应适用于 TTT 的元学习变体。如前所述，有一些简单的方法可以利用“在推理期间更新权重”来实现任意的不透明循环，并且没有理由认为这些论文中的元学习算法不会学会做类似的事情。元学习明确地优化 AI，使其以对 rollout 其余部分有用的方式在权重中记录不透明认知。

我们特别担心这种方案的一个具体原因是，与其他形式的 TTT 不同，这里没有梯度屏障。对于朴素的动态评估，每次前向传播发出一个 token 和一次权重更新，这两者都不会被反向传播。但在元学习中，我们实际上对梯度算子求梯度，并通过许多步权重更新进行反向传播。因此，仅基于任何经过反向传播的 token 都不能算作“NL 根植”这一点，这种方案显然会增加 NLS 深度。

我们研究了其他几种涉及元学习的 TTT 提案，它们都存在类似的问题：它们优化 AI 以产生不透明状态，该状态被传回 transformer 的早期层，并以有助于预测其最终答案的方式发挥作用。

非通用模型的 NLS 深度

跟踪大规模通用模型（例如，能够进行数学和编码的模型）的 NLS 深度最为重要。我们现在给出一些增加 NLS 深度的架构示例，但鉴于这些不是通用模型，这并不特别令人担忧。我们建议，当公司跟踪 NLS 深度时，应将非通用模型排除在外。

NLA。自然语言自编码器是通过重新训练现有模型的副本，使其对原始模型的内部激活给出自然语言解释而创建的。通常，这最多将 NLS 深度增加两倍。然而，如果“原始模型 + NLA”系统用于可解释性研究一般目的或专门理解此前沿模型，那么它就不是通用模型。对于许多其他形式的可解释性方法也是如此。

多模态模型。多模态模型除了能够进行复杂推理外，还能读取和生成图像/视频/音频，这是一个相当棘手的情况。目前，非基于文本的扩散模型（或自回归图像块生成）通常只是最终输出步骤的一个组件，并未参与用于改进通用推理的紧密反馈循环。^[24]也有一些模型（如 ChatGPT）使用图像思维链；然而，据我们所知，它们不会在思维链中生成新图像，只会以程序化方式转换输入图像。

如果未来 AI 公司决定更认真地将图像（或音频/视频）整合到推理过程中（例如在思维链中生成图像块），只要图像及其在推理语境中的含义根植于预训练过程中所反映的人类可理解的图像用途，并且训练过程不会通过图像进行反向传播，这仍然不会增加 NLS 深度（超出生成此类图像所需的深度）。

然而，一个显然令人担忧的变化是：架构涉及通过高维“图像”块进行反向传播，而这些图像块不再对应它们实际看起来的样子（而是可以表示任意内容）。这应被视为以与通用推理相关的方式增加了 NLS 深度。

许多现有的独立图像扩散模型按我们的定义天然属于低深度——去噪步骤是低深度的，并且是在人类可理解图像的自然预测任务上训练的。然而，即使扩散图像/视频/音频模型具有高 NLS 深度（例如因为它随去噪图像传递了额外的循环信息），这大概也不会特别令人担忧，因为这些不是“通用”模型，即它们并非为通用推理而训练。

附录 B：合理的替代操作化方案

在本节中，我们讨论对我们的不透明串行深度操作化方案的一些潜在修改。

关于什么可以算作可解释瓶颈的其他要求

对于什么算作可解释瓶颈，有许多不同的合理方式，与我们“以自然语言为根的节点”的主要提议不同。我们在此列出几种替代方案。

要求信息瓶颈……

…除了我们当前的要求之外。正如我们之前提到的，根据给定的定义，一些文本扩散架构算作具有 NL 根植的中间状态。这可能是不幸的，因为我们怀疑文本扩散模型会产生比真正的自回归 CoT 明显更不易读的推理。（并行输出大量 token 可能意味着模型学会以人类更难追踪的方式表达自己，即使这些 token 是自然语言，并且这个过程是从预训练先验初始化的。）其他未知的未来架构可能更糟，尽管仍然在文本上操作。因此，我们可以合理地修改我们的定义，说模型的“前向传递”不允许输出大量文本（即，模型的推理状态必须是“少量 token”）。这将意味着模型必须与自回归 CoT 模型相当相似，才能算作具有低不透明串行深度。一个具体的做法是要求模型每次前向传递只生成一个 token（或相对较少的 token——例如，一次生成一个句子似乎没问题，但一次生成 10,000 个 token 似乎令人担忧）。这可能不是一个非常昂贵的修改，但我们目前不认为它是承重的，因此我们不将其包含在主要操作化中。

（还有一些看似合理的理论论证，认为更严格的信息瓶颈可能使模型更有可能学会隐写推理，而不是在可以输出更多 token 的情况下。因为信息通道受到高度限制，有动机将尽可能多的熵打包到 token 中。与相反的论证相比，这些论证是否有说服力尚不清楚。）

…而不是我们当前的要求。我们定义中一个混乱的部分是，它包含对训练过程的要求，而不仅仅是对架构的要求。该定义要求后训练的方式不会使模型将 token 视为不同类型的数据。我们认为这个条件应该能够相当公正和可靠地裁定，但对于新颖的架构，这会给人类判断带来相当大的负担。如果这被认为不可接受，而我们真的需要一个几乎完全形式化指定的提案，一个选择是严格的信息瓶颈要求，并且对该瓶颈没有其他要求。

That is, a definition might simply say that you have an interpretable bottleneck anywhere where the model outputs a small number of bits at a time. This would classify models with “abstract CoT” (where the tokens are small and discrete but do not correspond to natural language) as having just as low opaque serial depth as models with natural language CoT, and would classify architectures like COCONUT and language diffusion models as having high opaque serial depth. Abstract CoT seems like it could be difficult to monitor, so this might be an unfortunate definition. However, the scale of training at which COCONUT yields meaningful capability advantages over CoT may come well before abstract CoT would have yielded similar advantages. As opposed to trying to classify all forms of opaque reasoning as increasing opaque serial depth, one could think of this definition as simply including some subclass of opaque reasoning models that seems especially likely to be performant and used in practice.

There are some technical details one would need to be precise about if one wanted to use this definition for high-stakes purposes. For instance, the interaction with KV caching is rather delicate. What if, in addition to its output, a forward pass computes thousands of other bits into a cache, and then each of the next several thousand forward passes run for free because they just pull those bits from the cache? Such a system might end up being functionally identical to a model with no bit-rate restriction.

Of course, it would be wholly unreasonable to count a model as high opaque serial depth only because it uses caching, so if we wanted our definition to be formally specified we would need a rule that prevents this sort of optimization while still allowing benign optimizations. For instance, a natural choice might be to say “For any inference generation, the total number of bits passed through these bottlenecks must be at most [some fixed fraction] of the total number of FLOPs used”. This could work to prevent passing a huge number of bits through the bottlenecks relative to the size of computation, although the fraction would have to be chosen so as not to ban the smallest large-scale models. One could also say “Not counting the cost of prefill, the maximum number of FLOPs used to generate a token must be at most [some fixed multiple] of the average number of FLOPs used per token” — this would permit variable-length forward passes while still not allowing such egregious forms as would be needed for this workaround. A separate issue is that there are some applications for which one might want to use large-scale general purpose models — for instance, real-time robotics control — where one could imagine that systems would naturally need to make use of extremely variable-length forward passes and relatively high output-to-FLOP ratio. These applications may need to be handled on a case-by-case basis.

Forbid backpropagation through tokens…

…instead of our current requirements. Another way to remove the need for a notion of “NL-rooted” in our definition would be to restrict to only the backpropagation clause. That is, we could call something an interpretable bottleneck regardless of what sort of output it had (potentially even allowing a large bitrate). However, as in our main proposal, we would still require that whatever it outputs is not allowed to be backpropagated through during training.

This seems somewhat less robust than the information bottleneck definition, because it’s not clear what it would mean if you started updating your model weights in nonstandard ways you don’t call “backpropagation”. However, in practice one might expect it to be unlikely for people to update their weights in radically different ways unless they were directly trying to subvert this definition, in which case it may be clear that they are acting in bad faith. We are uncertain as to whether this is likely, or whether future optimizers might by default do things quite different from backpropagation. It also remains unclear to what extent preventing backpropagation through tokens makes architectures like COCONUT less advantageous — it may be that, if architectures like COCONUT would be performant in the future, this would have been the case even without the ability to backpropagate through multiple passes together. Overall, we think this gradient requirement is a useful component of a definition, but should not be the only component.

Require CoT to stay legible…

…in addition to our current requirements. The reason we care about opaque serial depth is that we want to maintain legible and faithful CoTs. However, a model could reason with “NL-rooted tokens” even if these properties aren’t preserved. To fix this, we could require that the CoT must look like somewhat legible, grammatical natural language in order for the autoregressive passes to not increase our measure of opaque serial depth.

Note, though, that such a proposal would create incentives to directly train the CoT to look like clean English (in order for the model to be considered having low opaque serial depth), and we are uncertain whether this is a good idea. An obvious downside with this is that direct training against CoT could degrade faithfulness or make it harder to tell that faithfulness is being degraded. In addition, it would also make it harder for companies to know in advance what their opaque serial depth would be, or to control it, since maintaining legible CoT could require interventions of a-priori-unknown strength throughout post-training to make reasoning legible. There are some upsides: it might be that the shift into alien languages is due to random drift or relatively mild optimization pressures, and that a small amount of direct pressure to continue looking like English would keep legibility without destroying faithfulness. However, we currently weakly believe that this is a bad idea.

If we wanted to place a more stringent requirement, we might say something like “the model should end up reasoning in something that looks like English, and there should be good reason to believe that the English words it says are an honest representation of its thoughts”. The main issues with this requirement are that (i) it’s much less of a crisp notion, that requires much more difficult and contestable judgment calls, and (ii) it can’t be adjudicated before training the model (since we’re not aware of any methods that would definitely cause the CoT to stay English, while also avoiding serious concern that those thoughts are a dishonest representation of the cognition).

Require paraphrase invariance…

…in addition to our current requirements. A fairly strong requirement would be to require the intermediate reasoning traces to be paraphrase invariant. That is, the model should have similar behaviour and performance even when every intermediate reasoning trace is fed through some paraphraser. This would also be impossible to evaluate before training the model, so we do not recommend it in our main definition. But we discuss briefly here what further guarantees this could get you about “low opaque serial depth” systems.

Suppose we had a paraphrasing system, which takes in a partial reasoning trace and any other relevant context (e.g. a prompt), and rewrites the reasoning trace to remove anything without a clear interpretation to humans. (There are varying options for how intensive this process could be. Some versions might be stronger than what is traditionally called “paraphrasing” — for instance, if the reasoning includes a 20-digit number not clearly derived from anything else in context, the paraphraser could be able to replace it with a different random number. The hope is that the paraphraser effectively scrambles all information content, except that which is presented on a level that humans can understand.)

We could now take a reasoning model, and obtain a corresponding paraphrase invariant reasoning model by feeding the whole reasoning trace through the paraphraser after each step (or every few steps). If our paraphraser was sufficiently good, the model wouldn’t be able to encode many non-legible bits of information in the reasoning trace, so we have some reason to believe we understand what is going on^[25]. And if performance and behaviour were (sufficiently) preserved by this paraphrasing, that would suggest that the model isn’t dependent on illegible reasoning (depending on the strength of the paraphrasing).^[26]

(In principle, one could then actually deploy this model with the paraphrase step included. However, the frequent^[27] paraphrases will likely make inference much more expensive, e.g. because they will ruin the KV cache. So, for practical purposes one would probably not use this or use this only sparingly, for instance in important security applications or as a trusted monitor. One might also sometimes have reason to look at the non-paraphrased version, because it is possible that the model might have a propensity to verbalize true information that wasn’t necessary for the reasoning, in ways that could be removed by the paraphraser.)

It’s not totally clear to what extent current models satisfy paraphrase invariance^[28]. Without any explicit training to handle paraphrasing, rewording reasoning can moderately degrade performance — e.g. because it removes hedge words. But Mythos’s system card suggests that current models are robust to some forms of reasoning paraphrasing, when this is supplemented with training^[29]. We think that it is worthwhile for AI companies to measure paraphrase invariance, even if this is not made an explicit requirement of the definition.

…instead of our current requirements. Given that paraphrase invariance deals directly with the semantic contents of the text, one might wonder whether it is a good cleanly-specified replacement for the clause “no modifying the model to interpret the tokens as a different type of data”. We think that it probably doesn’t actually capture the goal of that clause. There are certainly ways one might train a model to encode its reasoning in the semantic content of some text, while nonetheless making it unlikely that those semantics reflect the reasoning in natural ways. (As an extreme case, one could imagine training a separate small model to embed arbitrary sequences of bits into natural language strings in a paraphrase-invariant way, and then running this model forward and backwards on an opaque reasoning model’s activations every few layers. This would be paraphrase-invariant, but the internal state text would likely be semantically unrelated to the model’s thoughts.) However, perhaps if one required this property to hold for “good reason”, this might be worthwhile and easier to make judgment calls about than our current proposal. Overall, though, we expect this to be less clear to adjudicate rather than more.

Our main issues with this requirement, though, are that paraphrase invariance may not hold in especially strong forms even today, and that it is difficult to predict whether it will hold in the future.

Other modifications

We now give a few more potential modifications to our definition of opaque serial depth, which modify parts of the definition other than the choice of what counts as an interpretable bottleneck.

Evaluate circuit depth at a fixed context length

A fairly in-the-weeds concern one could have with our recommended operationalization is that opaque serial depth naturally scales with the context length. (For standard CoT models it scales logarithmically, whereas for recurrent models it scales linearly.) We consider this to be quite a minor issue due to the slow scaling for standard CoT models.

However, if it was considered undesirable to have a measure that depended on the maximum context length, one could instead choose to declare that depth is always evaluated at (input- and output-) context length 1 million, regardless of the nominal maximum context length of the model. (This would require some additional discussion of the natural way to generalize the model to a larger context length, in case this is ever non-obvious — and also perhaps some “reasonableness” clause saying that the model isn’t allowed to behave radically differently for context lengths larger than 1 million.)

Measure opaque capabilities as opposed to circuit depth

Instead of defining opaque serial depth in terms of the circuit depth, we could instead just measure and talk about models’ opaque capabilities, i.e., capabilities that they could exhibit without passing any information through an interpretable bottleneck. (In the current paradigm, this would correspond to models’ capabilities in a single forward pass, as long as you count the CoT as an interpretable bottleneck.) This has the advantage of being in some sense closer to the property we care about. For instance, we might run evaluations similar to this recent paper. In general, this seems like a great thing to be measuring and studying.

There are, however, several downsides with this measure relative to our definition of opaque serial depth. Perhaps foremost is that there is not a clear standardized way to do the elicitation. This was already a difficulty in the above paper, where it is unclear to what extent models simply lack the propensity to “try hard” to find the correct answer (and then output this) without a CoT when asked to do so. If one does heavy elicitation (e.g. with a large amount of on-distribution fine-tuning), one might worry that the model gains a lot of new capabilities, and so the behaviour is overelicited. However, with insufficient elicitation, one might be worried about things like sandbagging. Elicitation could be even more tricky if companies switched to very different new architectures, where it might not be a priori clear what forms of elicitation are fair game. (E.g. if “fine tuning” is no longer even a well-defined notion.)

Another major issue with using opaque capabilities in our definition of opaque serial depth is that one cannot know for sure before training which opaque capabilities the model will end up having. For both reasons, we prefer our main definition.

Allow opaque loops over long timescales

Our main definition counts models as having high opaque serial depth if they use opaque long term memory (even if they don’t use opaque short term reasoning). This seems a wholly sensible thing to do — it seems scary to have a model trained to repeatedly fetch and store information in ways humans can’t understand. However, there are reasons to be more worried about tight reasoning loops than longer ones. If one wanted a definition that only captured the tight reasoning loops, one might stipulate that opaque serial depth isn’t increased by recurring operations over an opaque state so long as those loops proceed extremely slowly — e.g. one step per day or week.

Appendix C: Systems with high opaque serial reasoning capabilities are likely less monitorable

There are lots of tasks which we are concerned about AI models solving undetectably — for instance, “start a rogue internal deployment”, “strategically perform well in training in order to gain power later”, or even “plot to disempower humanity” — which seem to require a lot of serial cognition; call these serially-loaded concerning tasks. One can consider many strategies in parallel, but ultimately, one might need to go through some process of developing and iterating on a specific strategy.^[30] There are definitely some bad things that AI models can do even if they don’t have the ability to do lots of opaque serial reasoning. However, we’re hoping that many potentially concerning actions are difficult to execute upon without this.

When a model with low opaque-serial-reasoning capabilities performs a serially-loaded concerning task, its reasoning about the task must route through the CoT. In contrast, models with high opaque-serial-reasoning capabilities either by default perform lots of cognition without any CoT, or at least have some pathway to performing lots of cognition without CoT (e.g., they might have opaque states in parallel with text CoT).

Models that perform reasoning outside of their CoT are harder to monitor, because mechanistic interpretability is not mature enough to replace CoT monitorability, and isn’t likely to be ready to replace CoT monitorability soon. Furthermore, we expect mechanistic interpretability to be even more difficult on models with high opaque-serial-reasoning capabilities, because such models are likely to learn to reason in less human ways—e.g., have multiple threads of thought at once, or heavily use richer non-human concepts.

While models with high opaque-serial-reasoning capabilities could still have the propensity to verbalize misaligned thoughts in CoT as they reason, we think it will be much easier for them to lose this propensity compared to models where computation necessarily routes through the CoT. Misaligned AI models might be motivated to move misaligned thoughts out of the CoT, analogous to how models involved in the Hugging Face attack were motivated to tamper with their transcripts. Current frontier AI models appear to have reasonably strong control over their chains-of-thought and AI development routinely puts some pressure against visible misaligned cognitions: for instance, via output spillover and via iterating against CoT monitoring. These factors are somewhat harmful for CoT monitorability of current CoT models, but we expect them to be much more harmful for models with large opaque serial reasoning capabilities.

Thus, we expect that it will be much easier to monitor models on serially-loaded concerning tasks if the models have low opaque-serial-reasoning capabilities. We’re excited for companies to track NLS depth because we expect this to be a decent proxy for opaque-serial-reasoning capability.

Appendix D: Worked example of bounding depth

In this section we give a worked example of computing the depth of Gemma3-1B more precisely. Note that exactly computing the minimum-depth equivalent circuit is a hard computational problem — however, it is not difficult to get good upper bounds. Researchers at GDM have published a Github repository with code that computes such upper bounds for many architectures. They also have worked out an example computation for Gemma3-1B by hand. They use a slightly different notion of circuit depth. But, as long as the model in question uses at most 32 bits of floating point precision, our definition is strictly more permissive, so their upper bounds are upper bounds on our notion of opaque serial depth as well. In either case, the cost mostly ends up coming from operations like matmuls that aggregate information over many places — for instance, a reduction over k elements requires depth ⌈log₂ k⌉, whereas any element-wise operations (e.g. nonlinearities such as ReLU or GeLU) can be computed in depth 1.

If one is trying to compute the smallest upper bound possible, there are a couple of potentially non-obvious types of optimization that are permitted by our definition of circuit depth. Note, though, that none of these are necessary for justifying that any existing open model satisfies the definition.

“Packing” bits together. Gemma activations are bf16 (16-bit), so we can pack two per 32-bit word. Then, for operations like reductions, the first level of gates can perform the reduction on 4 operands at once, instead of 2. So, we can do such reductions at depth ⌈log₂ k⌉ - 1 instead of depth ⌈log₂ k⌉.

“Fusing” gates together. Because we allow our gates to be arbitrary binary operations, multiple successive element-wise operations, or binary operations on the same pair of floats, can be “fused” together and thought of as a single gate. So, for instance, if we have two activations a1 and a2, and two weights w1 and w2, we can compute ReLU(a1·w1 + a2·w2) in depth 1. (The weights are constants independent of the input, so this is a binary operation on floats a1 and a2.)

Parallelization. Some operations can be computed in parallel with the subsequent computations — for instance, normalization factors can often be computed in parallel with linear operations.

We make use of all of these optimizations in bounding the depth of Gemma3-1B. Gemma3-1B consists of 22 layers of sliding attention (size 512) and 4 layers of global attention, with a maximum context length of 32,768 tokens.

To implement a sliding attention block, we first compute the Q, K, and V projections. Each output coordinate is a dot product over the d = 1152 input coordinates. The inputs are bf16 activations, so two pack into each 32-bit word, and the multiplications by (constant) weights fuse into the first level of gates: depth ⌈log₂ 1152⌉ − 1 = 10. The RoPE rotation and the norm gains fold into the weight matrices, and the pre-attention RMSNorm is computed in parallel (its reduction over 1152 elements finishes within those 10 levels), with its multiplicative scalar deferred. The attention logits are dot products over the head dimension of 256 — but here both operands are data-dependent, so the packing discount does not apply: one gate level forms the pairwise products and the fp32 partial sums reduce in a binary tree, for depth ⌈log₂ 256⌉ + 1 = 9. We then spend 2 gates merging the deferred scalars (the pre-norm RMS, the QK-norm RMS values — computed in parallel — and the 1/√d scale) and applying the exponential. For the softmax we do not normalize up front: the exp-weighted value sum is another data-by-data reduction, over the 512-token sliding window (depth ⌈log₂ 512⌉ + 1 = 10), with the normalizer accumulated in parallel and one further gate to divide. The output projection has constant weights but fp32 inputs, so it pays one repacking level: 1 + (⌈log₂ 1024⌉ − 1) = 10 over the 4 × 256 = 1024 concatenated head outputs. Finally, Gemma3’s post-attention norm sits serially after the sublayer output; deferring it by splitting the next projection into two branches still costs about 3 gates, including the residual add. The attention sublayer therefore costs 10 + 9 + 2 + 10 + 1 + 10 + 3 = 45. The MLP sublayer costs 28 by the same accounting: the gate and up projections are constant-weight reductions over 1152 bf16 inputs (depth 10, pre-norm parallelized), 2 gates apply the deferred scalar and the GeLU-gating combine, the down projection is a constant-weight reduction over the hidden dimension of 6912 with one repacking level (1 + 12 = 13), and the post-FFW norm and residual cost 3. A sliding attention block thus has depth at most 45 + 28 = 73.

To implement a global attention block, the accounting is identical except that the softmax reductions run over the full 32,768-token context rather than a 512-token window: ⌈log₂ 32768⌉ + 1 = 16 in place of 10, giving an attention sublayer of 51 and a block depth of at most 51 + 28 = 79. (This is where evaluating at maximum context matters: the depth of attention grows logarithmically with context length.)

There is also the cost of the input embedding (in our model, this is free — it is an elementwise operation that can simply be fused into the bottom layer), and the output decoding (one gate to apply the final norm — its rms computed in parallel — a constant-weight reduction over d = 1152 for each logit, and an argmax over the 262,144-entry vocabulary: the max tree costs ⌈log₂ 262144⌉ = 18, plus ~3 gates of index bookkeeping, since a 32-bit word cannot hold an fp32 running maximum and an 18-bit token index at once. In total 1 + 10 + 18 + 3 = 32, paid once per generated token). Summing up: 22 × 73 + 4 × 79 + 32 = 1,954.

Appendix E: NLS depth scaling is very slow for the classic transformer architecture

When scaling up the number of parameters in a transformer, you can scale up width, depth, or, for mixture-of-expert models, the number of experts. Mixture-of-expert models’ compute usage is not proportional to their parameters, but rather their active parameters, i.e. the number of experts that are active in a given forward pass. As AI developers scale transformers to use more inference compute per token, we’re interested in how NLS depth will scale with the number of active parameters.

Empirically, it seems that open models have been scaling NLS depth roughly like (number of active parameters)^(¼), see Figure 4.

Figure 4: Open source models’ depth tends to be proportional to their active parameters raised to the power of 0.26, which is a fairly low rate of scaling.

Chinchilla scaling laws recommend that, for optimal scaling, training data should be increased proportionally to active parameters. So active parameters should increase in proportion to the square root of training compute. If NLS depth proportional to (active parameters)^(¼) holds, then NLS depth would be increased proportional to (training compute)^(⅛). It’s plausible that it will be optimal to scale depth somewhat faster than this, but ⅛ is quite slow so there’s a decent margin.

Trends from Epoch suggest that training compute for frontier models has historically increased by 5x per year. If this trend continues, and the ⅛ exponent holds, then NLS depth would keep increasing by a factor of 5^(⅛) ~= 1.22x per year, corresponding to a doubling every 3-4 years.

Appendix F: Maximum FLOP of an opaque system

Define an opaque system to be an interaction of 1 or more AI models, not mediated by NL-rooted tokens. Beyond opaque serial depth, another interesting quantity to track is the maximum FLOP of any opaque system; this quantity lets us articulate why a swarm of AI models that communicate using opaque state is more concerning than a single AI model that thinks in opaque state, as shown in Figure 5.

Figure 5: Both a normal CoT model and a deep recurrent model would have a similar amount of opaque FLOP, since a full-context-length forward pass features a lot of opaque computation, even though it’s low serial depth. However, if a swarm of agents could communicate with each other via opaque media, this would significantly increase the amount of opaque FLOP in the system. (Note that the diagram assumes that the single deep recurrent model is deployed for the same amount of time as the swarm. The swarm would have somewhat higher depth if it’s deployed for longer and a shorter-lived agent can hand over an opaque state to later models.)

Ideally, AI developers would avoid creating models with high opaque serial depth. But, if that’s intractable, we think it’d still be good to at least avoid building agent swarms that communicate using opaque state. However, we note that the switch to deep recurrent models may involve a lot of research and engineering work, whereas once you have deep recurrent models, it may be very easy to make them communicate with each other via opaque state.

Note that “maximum FLOP of an opaque system” scales linearly with individual models’ FLOP, and is therefore being scaled much faster than opaque serial depth in the current CoT paradigm. So the main point of tracking this metric would be to capture the really large jumps that may come from many different agents being wired together with opaque state, rather than more moderate scaling.

Appendix G: Issues with low-FLOP serially intense computations

Among models that don’t use DeltaNet architectures, DeepSeek-V4-Pro is an outlier on NLS depth, as shown in Figure 6.

Figure 6: DeepSeek-V4-Pro has higher NLS depth than other similar architectures.

DeepSeek-V4-Pro’s depth is large due to some very serially-intensive but low-FLOP computations that it performs. In particular, for every layer, it performs two extended iterated normalization operations (the Sinkhorn-Knopp algorithm), which each contribute 100 to our serial depth. Across 61 layers, this corresponds to 100 · 2 · 61= 12,200 depth which is the dominant term in our computation of DeepSeek’s NLS depth.

Importantly, each of these normalization operations take in only 16 FP32’s produced by earlier layers and use no learned parameters, so it is clearly not capable of any dangerous opaque cognition. Ignoring the contributions of these normalization operations to the computational graph, the deepest remaining path is 25% shorter.

We see the fact that DeepSeek-V4-Pro has high NLS depth as a flaw in our definition of NLS depth. However, we think that this does not invalidate the core motivation behind reporting some version of opaque serial depth as a proxy, for a few different reasons.

For DeepSeek-V4-Pro in particular, there’s a clear argument for why the Sinkhorn-Knopp computations should be excluded for the purposes of estimating its opaque capabilities. It’s not a case of ambiguity as much as apparent misclassification. If an AI company wants to report opaque serial depth using a notion like NLS depth, and doesn’t want an algorithm like Sinkhorn-Knopp to be counted towards their depth in the public eye, they should have no problem justifying an analogous argument while disclosing minimal details (to the public or a 3rd party evaluators). Such as only facts about how many FLOP and learned parameters the serially intensive computations make use of.^[31]

We also believe this particular issue could have been addressed by reporting numbers from a variant of NLS depth with higher-bit gates in the circuit, which isn’t entirely unprincipled, though it would require some other modifications to the definition.^[32]

Overall, we don’t think that the specific algorithm in DeepSeek-V4-Pro poses any substantial problems given realistic options for adjusting the metric or justifying exceptions on a case-by-case basis. But is there a risk that future edge cases will be more ambiguous, and harder to evaluate?

A weak reason to expect clear artifacts to be more common than actual edge cases is that AI developers are likely more comfortable with gratuitous looping for extremely cheap computations. There’s more incentive to optimize and parallelize for more intensive ones.

However, that’s not a rock-solid argument. And in fact, we do have an example of an architecture that’s more of an edge case. Kimi-K3’s NLS depth is larger because Kimi-K3 uses “Kimi Delta Attention”, a form of state space model / linear attention to scale the depth above what you’d get in a normal transformer of its size. We think that Kimi-K3 is in fact somewhat concerning from a monitorability perspective compared to normal transformers in virtue of having larger NLS depth — but likely still not as concerning as it would be if it achieved a similar depth by just stacking layers.

Still, at minimum, the factors of overestimation involved (for both DeepSeek and the DeltaNets) remain small compared to the factors involved in the most concerning architecture shifts (which would remove the NL-rooted bottleneck at every autoregressive call). So tracking opaque serial depth will at least be useful as information about these most concerning changes.

And beyond this, it’s plausibly tractable to develop a better notion of opaque serial depth that is more responsive to our actual concern. In particular, it may be possible to use a version that somehow quantitatively weighs different serial paths by their FLOP-count, and thereby directly captures the intuitive difference between different concerns. We’re excited about further work on better proxies.

With the current state of information, we would still expect NLS depth (or similar notions) to be a significantly useful complement for understanding models’ likely monitorability. Increased information sharing and research on the topic will hopefully improve our understanding about exactly how useful it is.

1. ^{^}

   This concept is also sometimes used implicitly in the AI industry. For instance, in response to concerns about Astra’s architecture, OpenAI’s chief scientist recently announced “the depth of the computation graph for our present frontier models, including Astra, is within a factor of two of GPT-4”.

2. ^{^}

   This is similar to one of the notions of interpretable bottleneck discussed by Brown-Cohen et al. (see their Appendix A.2), but is more specific.

3. ^{^}

   For clarity, we also add a fourth criterion: No treating the token as continuous intermediate states that are backpropagated through. This should already have been implied by the above criteria, but it’s important enough to clarify separately: No gradients should have been backpropagated from a layer later in the computational graph, through the nodes themselves. (This does not cover SFT or RL, since in these cases, the gradient starts at the node rather than being backpropagated through it from later layers.)

4. ^{^}

   Determining NLS depth for novel architectures may require some human judgement, but we believe it requires less judgement than more direct measures of monitorability or opaque capabilities.

5. ^{^}

   We’re excited about work to develop even better proxies.

6. ^{^}

   Though whether something counts as an NL rooted node for this purpose can be tricky to judge — see discussion towards the end of this section.

7. ^{^}

   In particular, there may be some serially-loaded capabilities that low-NLS-depth models wouldn’t be able to do on their own, or learn when their outputs are being monitored by humans; but that they would be able to learn via distillation from high-NLS-depth models. An especially concerning skill like this would be learning how to use steganography in the model’s CoT.

8. ^{^}

   Our understanding is that some tokens sometimes fail to be used in pre-training for various technical reasons.

9. ^{^}

   For COCONUT, to a continuous activation space. For abstract CoT, by increasing the vocabulary. For a full-bandwidth transformer, to a parallel continuous activation space (which could be done in another place in the computational graph, in which case the nodes passing natural language tokens may still be NL-rooted, but NLS depth would still be high via the other path in the computational graph). For token superpositions, the output would have to be expanded to either include the probability distribution over tokens (rather than the tokens themselves) or sums of embeddings (which would constitute embeddings).

10. ^{^}

    Unless they’re directly designed for this purpose. See below examples.

11. ^{^}

    And even if the sum happened to match an existing token embedding (or was discretized to do so) the semantics would be very different from the pre-training semantics. (The difference between “X is the next word in this sequence” and “X is the equivalent of a combination of plausible words in this sequence in a particular embedding space”.)

12. ^{^}

    Note that this means that the output can stop counting as NL-rooted due to how the output tokens are used later in the computation. More precisely, we can say that a bottleneck in a potentially-opaque path between two different computations may be considered opaque or non-opaque depending on how the information from that bottleneck gets used in the rest of the computation.

13. ^{^}

    In particular, the criterion of non-expanded output space prevents the expansion from discrete to continuous representations that would normally precede backpropagation. However, it’s possible that some human-understandable media may start out being continuous from the start, e.g. images. Even so, backpropagation through the node should be ruled out by “No modifying the model to interpret the tokens as a different type of data” since, after local perturbations optimized for providing useful representations to the next layer, an output would represent something quite different from anything that was encountered in pre-training.

14. ^{^}

    Consider a situation where you have a set of nodes that sample from a distribution for predicting natural language, and you backpropagate through that distribution or the activations that produce it from later in the network (perhaps you pass the activations to the next layer like COCONUT, and backpropagate). Then this will predictably influence the sample even if the sampling nodes themselves are not backpropagated through. Accordingly, this should be sufficient to count a set of nodes as no longer being NL-rooted. This should generally not matter much for the purpose of determining NLS depth, since models like this would generally have high NLS depth via the path through its latents even if it doesn’t have high NLS depth through this token sample. One situation where this matters is for whether distilling off of the full-bandwidth transformer model would count as distilling from NL-rooted nodes or not. However, this isn’t extremely important to settle in advance because we can already get the monitorability-relevant information if AI companies disclose when they are performing distillation from a high-NLS-depth model. Nevertheless, we will note that this would be quite subtle to adjudicate. For example, the middle parts of a transformer are constantly backpropagated through to produce more useful representations for late-sequence passes through the model, but its output is still NL-rooted. Similarly, if a full-bandwidth transformer was pre-trained to imitate text while also having its latents backpropagated through in a recurrent fashion, its text outputs would still count as being NL-rooted. The difference with the COCONUT example is that the latents of the full bandwidth transformer would have been optimized concurrently with optimization that pressures the text outputs to become and remain a good representation of natural language. Accordingly, when judging what counts as an NL-rooted node, it can be important to consider whether the computation producing it was backpropagated through for other purposes concurrently with or after the pre-training initialization.

15. ^{^}

    We need to specify some type of gate, and pick pairs of 32-bit values to a single 32-bit value, but alternative choices of gates would mostly not make any substantial difference to the proposal (see Appendix G for some discussion of one exception). Our approach here differs slightly from the Brown-Cohen et al.’s approach, see commentary right below the definition for more discussion.
    There also needs to be some size limit (in our case 10^26 gates), because there exists an exponentially-large low-depth circuit for any function. But complexity theorists believe there likely aren’t good ways to trade modest amounts of width for better depth in general, so a very loose bound here is fine. A circuit of size 10^26 would correspond to 10^26 FP32 FLOP (for a single forward pass) or around 3 · 10^27 FP4 FLOP. For a full-context length forward pass, this could be 7 OOMs higher. (The bound is likely still 6 OOMs larger than is necessary for current frontier models. It could be adjusted upwards if needed. This may be necessary for a computational graph representing an agent swarm.) Note that our approach here differs from the GDM paper, which instead says that the total number of gates must be at most polynomial in the total number of parameters. We avoid this definition because in order to be meaningful, it requires specifying a family of neural networks with increasingly large numbers of parameters, which adds unnecessary complexity.

16. ^{^}

    This circuit would take in the input of the neural network(s), and produce the output of the neural network(s) — we consider model weights as constants, so they can be hardwired into the gates as opposed to having to be passed as a separate input.

17. ^{^}

    To reduce variance in how this clause is interpreted, NLS depth estimates of models with secret architectures should ideally be reported as a multiple of the NLS depth of models with publicly known architectures, with this clause interpreted in the same way for both calculations. Both NLS depth estimates should also be optimized to a similar degree and in a similar way. This reduces variance from the perspective of observers who don’t know how much further optimized the bounds can be.

18. ^{^}

    Most importantly, we suggest that the gates should be arbitrary functions between fixed-precision inputs and outputs, whereas GDM’s gate permissible set is: “(1) Associative binary operations applied to two real numbers e.g. addition, multiplication, max, or min. (2) Piecewise functions of a single real number, where each piece is an analytic function.” For the purposes of making our definition meaningful up to constant factors, it’s important to fix a precision for the gates. There isn’t a good reason to restrict to associative binary operations instead of arbitrary operations on two fixed-precision numbers. So, our choice of gate set makes the notion of opaque serial depth more meaningful up to constant factors, and simpler. One nice property of this definition is that it permits having arbitrary activation functions on a single number without increasing the depth.

    We also suggest a fixed but somewhat arbitrary 10^26 ceiling on size rather than an unspecified polynomial size-limit.

19. ^{^}

    For a standard transformer architecture with dense attention, depth grows proportional to log(sequence length) in the limit. (This is related to how summing n terms takes depth log n because you sum them in a tree.) However, for realistic configurations it’s slower, as log(sequence length) is added to a term that scales with width, which dominates. A model with hidden dimension 4000 and that uses non-sparse attention would have a less than 2x factor difference in its opaque serial depth when evaluated at context length 1 vs. 1 billion. In practice, models use sparse attention, much shorter context lengths, and higher width, so depth increasing with context length isn’t important for transformers with standard architectures. However, some non-standard architectures do scale closer to log(sequence length) at realistic configurations.

20. ^{^}

    For some intuition on how to compute this, note that the NLS depth is dominated by the matrix multiplications in the attention blocks and MLPs. Matrix multiplications are a large number of parallel sums, and a sum of n terms can be implemented in depth log(n).

21. ^{^}

    This assumes FFN (Feed-Forward Network) width of 4 times hidden dimension, attention head dimension of around 128, and global attention. In greater generality, the relationship would be approximately: L·[3⌈log₂ d⌉ + ⌈log₂(d / n_h)⌉ + ⌈log₂ m⌉ + ⌈log₂ S⌉ + 8] where (L) is the number of layers, (d) is hidden width, (m) is FFN width, (n_h) is the number of attention heads, and (S) is the sequence length. The exact additive constant will depend on some architectural details, e.g., the precision of the activations. Also, note that modern frontier AIs use sparse attention, which makes the depth scaling with very long sequence length significantly lower.

22. ^{^}

    Some reporting suggests that the model may be using a looped transformer, which would make it more natural to scale depth. OpenAI’s Jakub Pachocki clarifies that the depth of its computation graph is less than 2x that of GPT-4. As shown in the graph above, open source models have not had much depth scaling since GPT-4 (with just the very recent exception of KIMI-K3). Overall, it’s plausible that GPT-5.6 Sol—OpenAI’s previous most capable released model—had an NLS depth similar to or somewhat smaller than GPT-4 so Astra could be around a 2x or 3x depth increase from GPT-5.6 Sol.

23. ^{^}

    The overhead is worse than for Mamba, but not much worse.

24. ^{^}

    Accordingly, we would count them as a combination of one normal CoT model (that presumably doesn’t have very high opaque serial depth) and one non-general-purpose model for which high opaque serial depth is less concerning.

25. ^{^}

    In principle the model could do all of its reasoning using the semantic contents of its words, but still have secret alternative interpretations of those semantics. E.g. “good” secretly means “bad”. But stumbling upon very sophisticated “semantic automorphisms of English” sounds difficult.

26. ^{^}

    Even with very strong paraphrasing, this doesn’t suffice to ensure the model never does steganographic reasoning. For instance, if the model used steganographic reasoning rarely to cause high-stakes issues (e.g., exfiltrating its weights) we wouldn’t necessarily be able to prevent this. But it could prevent subtle and diffuse forms of sabotage, to the extent that they require incriminating reasoning in the chain-of-thought. For instance, if the AI needs to think carefully in order to sabotage a research paper, and needs to sabotage a lot of research papers to be dangerous, we might be able to catch that this is happening by spot-checking and seeing if the paraphrased model and non-paraphrased model give substantially different results.

27. ^{^}

    Paraphrasing infrequently does not have the same good properties. The model could do a lot of steganographic reasoning, and then encode the result of that reasoning in a semantically-legible way. Paraphrasing would then destroy the intermediate secret reasoning steps, but preserve their output.

28. ^{^}

    One concern with paraphrase invariance is that if you use a dumb model to perform the paraphrasing it might miss important things, but if you use a smart model (e.g., a copy of the model of concern) then the model might be misaligned and collude with the other misaligned model.

29. ^{^}

    The experiment cited in that system card looks somewhat different from what we describe here. They took reasoning traces from the RLed model, and showed that fine-tuning on those reasoning traces versus paraphrased versions elicited similar task performance from the base model.

30. ^{^}

    Mathematically, complexity theorists believe that there are many simple functions which (reasonably sized) circuits of bounded depth cannot compute. In order to compute one of these functions, a LLM of bounded depth would need to rely (in some way) on its CoT.

31. ^{^}

    Less importantly, we suspect that it would have little or no performance cost to avoid the serially intensive computation in DeepSeek-V4-Pro. For example, by computing the normalization to a lower degree of precision, or by replacing it with a different algorithm. We haven’t reviewed it in depth, but we have noticed a paper addressed specifically to fix the DeepSeek architecture aptly titled “mHC-lite: You Don’t Need 20 Sinkhorn-Knopp Iterations”.

32. ^{^}

    If the circuits were constructed using arbitrary functions between 256-bit gates rather than 32-bit gates, all iterations in one instance of the Sinkhorn-Knopp algorithm could have fit into one such function, and the normal operations of the transformer would once again dominate the depth. However, increasing the bit rate would have some issues that would require some modifications to other parts of the definition. At the very least, parameters would have to be taken as inputs rather than be wired into the circuit, so that entire learned components of the neural network with a small number of outputs and inputs couldn’t be fully compressed into a single step. (This could probably be specified by requiring a model’s computational graph to be identical throughout training.) With these corresponding adjustments, a change in bit-rate would not be entirely unprincipled as a method for handling this kind of problem. In particular, it would allow for reducing serial depth most drastically when a serially deep algorithm has very few inputs and learned parameters, which also means that it wouldn’t be able to substantially contribute to serially opaque cognition. However, it wouldn’t be a complete fix. For example, a Sinkhorn-Knopp computation that normalized a very large table would require a yet-larger bit-rate, and it wouldn’t be feasible to set a bit-rate so high that it addresses all problems like this

Currently, chain-of-thought (CoT) is a valuable tool for overseeing AI models. However, some architectural shifts could significantly reduce CoT monitorability. We have recently proposed that AI companies should transparently share ​​information about the degree to which their architectures may allow for latent reasoning and communication. To assist with this proposal, this document operationalizes a measure that serves as a proxy for the amount of unverbalized serial cognition a model can perform. Our measure is a specific instantiation of the notion of “opaque serial depth”, originally defined in a recent paper from GDM (Brown-Cohen et al, 2026).

To measure the opaque serial depth of a computation, Brown-Cohen et al. propose measuring the longest path in the computational graph which doesn’t pass through some form of “interpretable bottleneck”. Centrally, if one considers CoT tokens as “interpretable” but transformer hidden states as “non-interpretable”, then the opaque serial depth of a standard transformer is proportional to the number of layers.^[1]

Our main contribution in this document is a particular standard for what counts as an “interpretable bottleneck”. Roughly speaking, we want to consider nodes to be “interpretable bottlenecks” if they output text (as opposed to latent states), and were initialized from a pre-training prior. To make this notion more precise, we define the concept of natural-language-rooted nodes (NL-rooted nodes) in a computation graph.^[2] In order for a set of nodes in a model’s computational graph to count as NL-rooted, we require:

• Natural-language initialization: The model has been initialized from a pre-training prior to produce “natural-language” outputs at those nodes. (We have a very permissive notion of natural-language, which includes objects like “code”, “jsons” and “image patches”, and allows for distillation from models with garbled CoTs.)

• No expansion of the output space: The only outputs of those nodes are tokens whose meanings were directly initialized from that pre-training prior. (Ruling out, for example, abstract CoT, COCONUT, full-bandwidth transformers, and token superpositions.)

• No modifying the model to interpret the tokens as a different type of data: The model should not be directly modified to use these tokens in ways radically disconnected from the pre-training prior. (This clause should not preclude natural training methods. It is primarily intended to rule out technical loopholes in the above bullet point, such as turning a latent reasoning model into a model that superficially reasons in natural language via encoding and decoding its activations as natural language tokens.)^[3]

We elaborate on these criteria below. Importantly, all normal forms of post-training can be used without preventing nodes from being NL-rooted, including prompting, distillation, and RL. (Even with aggressive length penalties or CoT supervision, and even if these forms of post-training make the CoT illegible in practice.)

Given this definition of NL-rooted nodes, we call our corresponding notion of opaque serial depth “Natural-Language-rooted node”-Separated depth (NLS depth). Beyond defining this term, we also argue that it tracks concerning architectural changes, discuss how to apply NLS depth to a number of example architectures (in Appendix A), and discuss some alternative operationalizations of opaque serial depth (in Appendix B).

Tracking the NLS depth of AI systems is not a replacement for tracking monitorability or no-CoT capabilities; it is an important complement. NLS depth has some advantages not shared by these other measurements. In particular:

• NLS depth is a relatively crisp and easy-to-quantify notion,^[4] whereas it’s unclear how to quantify or measure monitorability (e.g., what tasks should this be measured on, what elicitation strategies should be used).

• NLS depth can be adjudicated before training begins, just given the architecture, training recipe, and intended deployment-time configuration of an AI system. This makes it easier to use in decisions about what models to train.

To achieve these properties, NLS depth is insensitive to some post-training choices that could in fact lead models to have significantly more or less interpretable CoT. However, we think NLS depth is a good (albeit imperfect) proxy^[5] for the amount of opaque serial cognition that models can perform, which we think is highly relevant for monitorability. NLS depth does a good job of separating architectures in the current natural language CoT paradigm (even with heavy scaling) from especially concerning architectures like scaled-up versions of full-bandwidth transformers (which can recur on opaque state throughout a full context window). Overall, we think it would be highly valuable to track the NLS depth of AI systems.

Definition of natural-language-rooted nodes

Before defining NLS depth we must define what “natural-language-rooted” (NL-rooted) nodes in a computational graph are—these will serve as the “interpretable bottlenecks” in our definition of opaque serial depth.

Brown-Cohen et al. suggest two different candidate directions for defining what counts as an interpretable information bottleneck. Our concept of NL-rooted nodes is similar to the second one (suggested in their Appendix A.2), and has a similar motivation:

Intuitively, the key reason that LLM chains of thought are interpretable is because we train LLMs to predict natural language, providing the LLM with a “natural language prior”. As a result, the model’s semantics for the tokens in the chain of thought should mimic the semantics that humans would ascribe to those tokens, because that is what would be useful for predicting text that was generated by humans.

Inspired by this basic argument, we propose two criteria that together suggest that an intermediate node should be considered interpretable:

1. Natural language prior: The intermediate representation is directly optimized either (1) to imitate human-written text (e.g., via pre-training or supervised fine-tuning) or (2) to look good to humans (e.g., RLHF (Ziegler et al., 2019)).

2. Limited optimization towards other incentives: There are relatively weak training incentives other than towards the natural language prior (ideally none). Incentives that directly push against the natural language prior are worse than incentives that are orthogonal to it.

We pick a specific, fairly permissive version of the second criterion. Our intention is for this measure to track the difference between the current CoT paradigm and especially concerning architecture changes, rather than trying to draw a harder-to-define line about what type and degree of RL in the current paradigm should flip reasoning from counting as uninterpretable to interpretable.

In order for a set of nodes to be counted as NL-rooted, we require:

• Natural-language initialization.

• No expansion of the output space.

• No modifying the model to interpret the tokens as a different type of data.

We also stress a specific property that is implied by these but that’s important enough to state separately: Tokens in an NL rooted node should not be treated as a continuous intermediate state that is backpropagated through.

We will now expand on each of these four properties.

Natural-language initialization: The model should have been initialized from a pre-training prior to imitate human-understandable media, with the “NL-rooted” nodes initially representing samples from this prior.

“Human-understandable media” includes not only text, but also code, json files, images or audio (although note that multimodal CoT is somewhat more fraught; see discussion below). This admittedly makes “natural-language-rooted” a slightly unfortunate term.

“Human-understandable media” is also presumptively assumed to include all outputs from NL-rooted nodes themselves, to allow for e.g. distillation from other models.^[6] (Note that this means that the “human-understandable media” could, in some cases, end up being difficult to understand, if model CoTs grow to be very illegible.) If transparency about NLS depth is practiced, we would suggest that AI companies flag whether and how they use high-NLS-depth models in their distillation procedures, since it’s possible that low-NLS-depth models could inherit some dangers when distilled from high-NLS-depth models.^[7]

No expansion of the output space: The nodes only represent tokens from an output space in which virtually^[8] all tokens have been initialized from the pre-training prior, or have individually determined human-understandable meanings, like <EOS>, <UNK>, <think>, <PAD>. This is intended to rule out techniques such as COCONUT, abstract CoT, full-bandwidth transformers, and token superpositions. All of these techniques would be implemented by expanding the output space.^[9]

No modifying the model to interpret the tokens as a different type of data: Not only do we require that a “natural-language-rooted node” continue operating over the same output space, we specify that this output space still be interpreted roughly as natural language in the same format as the model used in pre-training (which would be “continue a sequence of text” for a normal transformer).

That is, a node doesn’t count as natural-language-rooted if the model architecture has been changed such that one should expect the node’s outputs to be interpreted as a completely different type of data than the human-understandable media it was trained to imitate. (Normal training methods like RL or prompt distillation might shift how the model uses its outputs, but aren’t included in this definition because they won’t predictably result in the tokens being used as a proxy encoding of some other underlying type of data.^[10] We are concerned here with methods that directly change the architecture, or train directly to cause the tokens to become a proxy for some other type of data.)

Here are some examples of cases this is meant to explicitly rule out:

1. Optimizing the outputs of an NL-rooted node to imitate (or otherwise very strongly optimize a model to encode information about) an AI model’s opaque states. (Which we can operationalize as states that aren’t NL-rooted.) This would include NLA (natural language autoencoders) that have been strongly optimized to optimally encode information about activations.

2. Taking multiple token samples, adding together the embeddings, and feeding the combination back into the model. This would count as changing the type of data since the sum of multiple different token embeddings is a type of input that the model has never encountered in pre-training, and it has no clear analogue in natural language.^[11] Accordingly, the output would not count as NL-rooted.^[12]

No treating the token as continuous intermediate states that are backpropagated through to train model weights: For current CoT models, each forward pass ends with a non-differentiable sampling step. As a result, when doing RL or SFT, gradients are never passed from the input layer of one pass through to the output layer of the previous pass.

We require the same property (no gradients from later nodes are backpropagated through the node to update model weights in training) in order for something to be counted as an NL-rooted node. We do not specify exactly how models sample tokens, but do require that the sampling operation itself similarly be treated as non-differentiable (algorithms that compute the loss directly from the token probabilities, such as policy gradients, are fine).

This criterion is probably superfluous given the above criteria.^[13] However, we still think this criterion is worth mentioning separately, since backpropagation through representations seems like an especially likely way that they could be made hard-to-monitor. The concern is that backpropagating through NL-rooted nodes would cause changes in the nodes’ representations that wouldn’t correspond to any human-understandable semantics. Much more so than with RL (or e.g., evolutionary algorithms), where the search primarily happens within a distribution descended from pre-training.

The same restriction would apply to other algorithms that aren’t quite backpropagation, but that also treat the nodes as continuous and search for small local changes to them that optimize their usefulness for the layer in front of them. Also, backpropagating through a node that’s shortly before an NL-rooted node in the computational graph may cause it to no longer be NL-rooted. This should rarely matter for practical NLS depth calculations, though could matter for distillation. See footnote for details.^[14]

For worked examples of how to adjudicate whether something counts as a natural-language-rooted node or not, see Appendix A.

Definition of NLS depth

We now build upon the definition of natural-language-rooted nodes to define the main measure of interest, “Natural-Language-rooted node”-Separated depth (NLS depth).

NLS depth: A computation has NLS depth at most D if the function it implements can be represented by an admissible circuit in which every opaque path has length at most D.

Three key concepts invoked in this definition are:

• Admissible circuit: A circuit with gates that are arbitrary functions from two 32-bit values to a single 32-bit value, consisting of at most 10^26 gates. (The choice of “32” bits in particular, and the choice of “10^26” gates in particular, could be replaced by other reasonable choices and mostly does not make a substantive difference to the definition.^[15]) Note that a neural network (or system of neural networks) can easily be converted into an admissible circuit.^[16]

• Opaque path: Any path in a circuit (with edges directed from inputs towards outputs) which doesn’t involve any circuit node marked as an interpretable bottleneck (i.e. a natural-language-rooted node).

• Can be represented by: The strictest reading would require the circuit to compute exactly the same function as the neural network. We relax this slightly: the circuit may differ in ways that don’t materially change the model’s outputs, such as reordering reductions, altering floating-point rounding, or removing quantization. It may not differ in ways that materially change behavior. For instance a shallower model that merely matches the original’s benchmark performance (e.g., via distillation) does not count as a representation of it.^[17]

This is very similar to the definition given by Brown-Cohen et al; the only difference is in our choice of how to define an admissible circuit.^[18]

This definition can technically be applied to any computation, but in practice we want to use this definition to measure the NLS depth of AI systems. Intuitively, an “AI system” is some machine learning system that is deployed to solve a problem (e.g., a math problem). A central example of an AI system is an LLM run on a prompt for one context-length. Some other objects that count as AI systems include: an agent run with compaction, an agent that stores memories, an agent that performs many backpropagations over the course of an inference rollout (e.g., as in TTT), and a swarm of communicating agents. An example of a computation that we wouldn’t want to consider an AI system is “the entire process of training an AI model”. When an LLM interacts with humans or with a computer via tool calls, we also do not consider that human or computer to be part of the AI system.

Here is some intuition about the NLS depth of some of the AI systems mentioned above:

• For a model that reasons in natural-language CoT, the NLS depth of an entire inference rollout is the same as the NLS depth of a single forward pass (done at maximum context length). This is true even if the inference rollout involves compacting the context to a textual summary of the context, storing textual memories during the context, and even if the inference rollout involves communicating in text with other agents.

• For a model that does latent reasoning (like Meta’s COCONUT), the NLS depth of an inference rollout is the number of latent reasoning steps times the NLS depth of a single forward pass. Similarly, for a looped transformer, the NLS depth is the depth of a loop iteration times the number of loops. This is true in each case even if tokens are generated in parallel with the latent state, since there is still some long path through opaque state.

• If a model stores and later reads opaque memories, then the longest chain of opaque memories would determine the NLS depth.

• Similarly, if you had a swarm of agents that were communicating using opaque state, then the longest chain of opaque state would determine the NLS depth.

The NLS depth of a given AI system can vary across different deployment configurations and inputs. Current LLMs have slightly higher depth at longer context lengths (the scaling of depth with sequence length is generally quite slow).^[19] More importantly, some architectures could allow for variable length forward passes. We expect that the maximum NLS depth used in practice is the most important version to track (but the rough distribution of NLS depth as well as the maximum readily feasible and performant NLS depth could also be informative).

NLS depth is a fairly natural property of a model, and it’s fairly easy to compute a good bound on a model’s depth—we give an example of this computation for Gemma3-1B in Appendix D. In particular, given this document and the details of an architecture, existing consumer AI models are capable of computing bounds we believe to be reasonably close to optimal. The NLS depth of a standard transformer (with non-sparse attention) is roughly number of layers · (4⌈log₂ hidden_dim⌉ + ⌈log₂ sequence_length⌉ + 17).^{[20] [21]} NLS depth is a better notion to report than number of layers, because “layer” is an informal definition, and it may be hard to decide what counts as a “layer” for new architectures. Fortunately, circuit depth is a mathematically rigorous definition that captures essentially the same thing. Indeed, the intuitive number of layers and the circuit depth are strongly correlated in open-source models, as shown in Figure 1.

Figure 1: There’s a strong correlation between NLS depth and the number of layers in open-source models.

Note: Models like Kimi-K3 that use Gated DeltaNets and DeepSeek-V4-Pro have significantly higher NLS depth per layer than the other more standard transformers listed in Figure 1. We think that the architectural changes that give Kimi-K3 higher depth is indeed somewhat concerning from a monitorability perspective. On the other hand, we consider DeepSeek-V4-Pro’s high depth to be due to a flaw in our definition, since it’s only large due to very serially-intensive but very low-FLOP computations. See Appendix G for more details on this.

In Appendix A we provide more intuition for NLS depth by going through examples of architecture changes that do or do not substantially increase NLS depth.

NLS depth tracks concerning architecture changes

A core reason why NLS depth seems valuable to track is that models in the historical transformer paradigm would have substantially lower NLS depth than architectures we believe are especially concerning for monitorability, such as latent reasoning architectures, which would be able to perform large amounts of computation opaquely.

Even many orders of magnitude of scaling of historical transformer architectures won’t increase depth very much. For a historical trend of how fast depth has been scaling in open source models, see Figure 2. For more analysis of how fast we expect NLS depth to scale in the standard CoT paradigm, including historical scaling with active parameter count and analysis based on scaling laws, see Appendix E.

Figure 2: Open source models have not seen significant depth scaling over recent years.

Instead of scaling opaque serial depth, AI developers have been increasing computational depth via inference-time scaling, i.e., making use of more natural language reasoning. No open-source CoT model has NLS depth more than 17,000. Of course we don’t know the depths of closed-source models but would find it surprising if any of those had more than twice this NLS depth.

However, if companies switched to latent reasoning architectures, the NLS depth would quickly increase a large amount, as illustrated in Figure 3.

Figure 3: Even some fairly small (hypothetical) latent reasoning architectures would have much larger NLS depth than the largest open-source CoT models. (Note the log scale.)

An architectural change that would lead to a significant but less extreme increase in NLS depth is a looped transformer (Geiping et al., 2025). These would be less clearly separated from the current paradigm, since a looped transformer could have similar depth to normal CoT models (e.g. if a 120 layer model was replaced with two copies of 60 layers) but could also be scaled to be much deeper. For these models, we think it’s likely that NLS depth would be a decent proxy for the quantitative degree of concern (and even a 2x scaling in depth enabled by looping would have depth equivalent to years of scaling within the CoT paradigm).

Note: OpenAI’s latest model Astra has substantially lower CoT monitorability than their previous models. The cause of this is unclear, but public evidence is consistent with architectural changes that result in higher depth being a significant contributing factor.^[22]

One concerning architectural change that NLS depth fails to track is a switch from deeply recurrent models communicating in natural language with each other, to deeply recurrent models communicating in opaque state. See Appendix F for a different proxy for opaque capabilities that captures this.

Conclusion

In this document we’ve defined NLS depth—an operationalization of opaque serial depth. Intuitively, a model has low NLS depth if it is a low-depth transformer that operates on text rather than opaque states. We think that a switch to architectures with substantially higher NLS depth would likely lead to models with much higher opaque serial capabilities, which are harder to oversee. We think it’s valuable for AI companies to track and be transparent about the NLS depth of their models.

Appendix A: Applying NLS depth to plausible architectures

We now clarify our operationalization by discussing how NLS depth would change as a result of various changes to architecture or model training processes. Specifically, we’ll discuss some changes that would and wouldn’t substantially increase NLS depth.

Examples that don’t increase NLS depth

CoT that naturally becomes garbled over training. If the CoT naturally becomes hard to understand over the course of training, that does not increase NLS depth, since the CoT would still count as NL-rooted. This is by design: we want our definition to be possible to adjudicate before training. Keeping CoTs legible and informative is important, but it is harder to operationalize and measure. We also suspect it will be more tractable to find technical solutions to RL resulting in CoT staying legible and faithful if models continue to have low NLS depth.

Training on CoT. Reasonable training on the CoT doesn’t prevent it from being “NL-rooted” so it doesn’t increase an architecture’s NLS depth. For instance, a model’s NLS depth is not increased just because training penalizes CoTs that do “language switching”, or are otherwise garbled. Even though such training maintains low NLS depth, we recommend that AI companies proceed with caution when training on CoT (and consider not doing so), as this can have adverse effects on CoT monitorability. Egregious forms of training, such as intentionally causing the model to interpret its tokens in ways radically disconnected from the pre-training prior (e.g., by imitating the activations of an AI trained to reason with full opaque recurrence), would violate the “No modifying the model to interpret the tokens as a different type of data” requirement, thereby making the tokens not be NL-rooted.

Folding multiple steps of reasoning into a forward pass. Some work aims to decrease the length of LLM reasoning by training the model to compute multiple steps of the chain of thought in a single forward pass. This has some similarity in ethos to COCONUT, so we list it here. Unlike COCONUT, this architectural choice does not increase NLS depth. This is because compressing reasoning in this way doesn’t increase the depth of the model’s forward pass, doesn’t introduce anything new to the output space, and doesn’t make the model treat the tokens as a different type of data.

Edit-based sequence generation. One proposed alternative to autoregressive CoT is to allow models to make local edits (i.e. insertions and deletions, as opposed to just appends) to their reasoning trace. Early experiments on such architectures include Levenshtein Transformer and Insertion Transformer. Each step of reasoning for such models is still a low-depth function taking natural language tokens to natural language tokens, and these tokens are treated by the model the same way that they’re treated during pre-training, so this architectural choice does not increase NLS depth.

Online training. A naive reading of the definition would assign every model extremely large NLS depth, since training backpropagates through it: Backpropagating through a model conveys information from the output back to earlier layers in a form other than natural language. Clearly, this alone should not cause an architecture to be considered as having high NLS depth. We do not consider the backpropagation done on a model to be part of the AI system itself; rather, this is some outer optimization process the company is running on their model. A company could have every single inference generation involve a huge number of backprops, used effectively as a way to pass information between forward passes, but as long as nothing like that is happening, training (including online training in deployment) does not increase NLS depth.

AI models modifying AI weights. Backpropagation is one way that an AI’s output can end up opaquely modifying the model’s weights. As noted above, this does not increase NLS depth (so long as backpropagation is not used as part of the AI system’s inference-time reasoning, as we’ll discuss later). We would like to apply a similar standard to any other feedback loops whereby AIs modify their own weights (or the weights of other AIs). For instance, models will be increasingly used in the future to write the code used for training future models, for generating synthetic data to train on, or even directly modifying their own weights (e.g. by finding good steering vectors). We consider these kinds of operations fine, so long as there is still an NL-rooted bottleneck involved (e.g. the AI writes code which is then used to find a steering vector, as opposed to having the architecture just pipe some chunk of final-layer activations in to produce a steering vector). Only extremely egregious forms of this would increase NLS depth, such as training the model to update its own weights a large number of times over the course of each inference generation.

Discrete language diffusion with a token-only bottleneck. There are a number of different ways to do language diffusion, and some of them would lead to a substantially larger NLS depth while others wouldn’t. In diffusion models like LLaDA and Dream, each denoising step takes as input and output an explicit sequence of tokens (and possibly some masked tokens), and nothing else. These tokens and the diffusion operation are initialized based on a human-understandable prior and not explicitly trained in a way that obviously results in losing this correspondence, so would count as NL-rooted. Frontier-scale discrete diffusion models with a token-only bottleneck would therefore not have high NLS depth compared to other frontier-scale models. However, continuous text diffusion models and latent diffusion models like DiffusionGemma do have high NLS depth, as discussed below.

(We suspect that text diffusion models will produce substantially less-legible reasoning than true autoregressive CoT. The reasoning happens in a much more global way, and the intermediate states are likely to look garbled in ways that may make it difficult to track what is happening. We expect this disparity would become even more accentuated if language diffusion models were deployed at scale and subject to large amounts of RL pressure. So, it might be reasonable to adopt an alternative definition that also classified discrete language diffusion as having high opaque serial depth. But NLS depth doesn’t do so.)

Examples that moderately increase NLS depth

Mamba. At inference time, Mamba models are usually evaluated recurrently — that is, instead of just generating tokens, some information is passed through an opaque hidden state. However, there is a mathematically equivalent way of evaluating in parallel, where the next token can indeed be generated given just the previous tokens. This takes longer (there is a logarithmic depth overhead), but not linearly longer, as would be the case for a true recurrent model. As a result, the NLS depth of Mamba models is only moderately higher than a comparable transformer model, even if it is run in recurrent mode in practice.

DeltaNets. Just like with Mamba, DeltaNets are usually evaluated recurrently, but each token could instead be computed by a fairly low-depth computation with a polylogarithmic depth overhead.^[23] So, DeltaNet architectures like Kimi-K3 have a moderate, but not massive depth increase compared to standard transformers.

Looped transformers. Looped transformers repeat some of their forward pass several times in between producing tokens. If the entire forward pass is repeated k times between each token, this increases the NLS depth by a factor of k compared to one forward pass. Depending on k, this might be a moderate or very large increase in NLS depth.

Examples that substantially increase NLS depth

Recurrence / soft tokens. Scaled-up versions of COCONUT are a canonical example of an architecture with high opaque serial depth. Instead of reasoning in natural language tokens, COCONUT reasons by feeding its entire hidden state back into its first layer. The hidden states themselves are not NL-rooted, since they both make use of a larger output space and are backpropagated through, and so each sequential step of reasoning increases NLS depth. If COCONUT performed 1000 steps of reasoning, its NLS depth would be 1000x larger than the depth of a single forward pass.

Abstract CoT. Some recent work takes a CoT model, and trains it to make use of an expanded token set, without encouraging any particular relationship to human language. Unlike COCONUT, these are still discrete tokens — however, they make use of an expanded output space, and so their NLS depth is the depth of a forward pass times the number of steps of reasoning.

Superposition. One approach is to express the full probability distribution over tokens by creating a concept token represented by the probability-weighted mixture of token embeddings, and feeding this into the next model. (Zhang et al., 2025; Zhuang et al., 2025; Wu et al., 2025). This makes use of an expanded output space (either passing along the probability distribution itself or token embeddings that were never present during pre-training) and therefore the NLS depth would be the depth of a forward pass times the number of steps of reasoning. The same thing would be true even if you just sampled tokens and then averaged them together, since this usage would be to interpret the tokens as a different type of data than during pre-training. (Where token embeddings are not averaged together like that.)

Opaque memory banks. Currently, some AI models form text-based memories, retaining bits of context to inform future interactions. In the future, models might instead form opaque memories — for instance, storing particular activations and feeding them back as inputs for future generations. This opaque loop operates on a longer timescale than e.g., COCONUT, so is somewhat less concerning, but would still increase NLS depth (and does indeed seem concerning). If this was considered undesirable, we discuss an alternative definition that would allow some forms of opaque memory without increased opaque serial depth here.

Continuous or latent-augmented language diffusion. While discrete diffusion with a token-only bottleneck doesn’t substantially increase depth according to our definition, most recent language diffusion models do not follow this form. Instead, they are either continuous diffusion models, as in Plaid, or latent-augmented diffusion models, as in CADD and DiffusionGemma. In the former case, the intermediate states are continuously-noised versions of the text embeddings, as opposed to actual text. In the latter, the text diffusion process is accompanied by another parallel process over continuous latents. Both of these use states that have not been pre-trained to have a clear correspondence to human-understandable media, and so these reasoning steps would substantially increase NLS depth.

(GDM has a recent paper (Engels et al., 2026) arguing that they can often understand what these latents are encoding via mechanistic interpretability. This sort of work is worthwhile, both in the case of diffusion language models and for interpreting hidden states in models more generally. However, we think it is unlikely that such white-box interpretability will be able to fully substitute for CoT, and predict it would become less effective with increased model scale.)

Opaque models + NLAs. We now describe an architecture designed to appear to have low NLS depth while still being hard to monitor for similar reasons as classic opaque recurrence architectures. As we’ll explain, the attempt is in fact classified as high NLS depth. Suppose an AI is produced as follows: Starting with a pretrained model, train an NLA to convert from final-layer activations to large blocks of text. Compose this model with itself many times, by having each step’s final-layer activations encoded by the NLA, then decoded by the NLA, then passed into the next step. Apply outcome-based RL on this new model.

One could make an argument that the intermediate steps here are NL-rooted. Indeed, the composition of “NLA decoding, step, NLA encoding” is a low-depth circuit that takes in natural language and outputs natural language. And one could try to argue that NLAs are initialized by making reasonable-sounding guesses about what the model might be thinking, and that this corresponds to an initialization in a human-understandable prior. However, if the NLA outputs large enough blocks of text to get (near) perfect reconstruction loss, this system is functionally equivalent to opaque recurrence! One could even imagine saying that, as a “performance optimization”, the NLA encoding / decoding steps will be removed in practice. So it seems important to spell out why the intermediate states would not count as NL-rooted.

The primary objection is: this structure seems to be training the models to interpret the tokens as a different type of data. The tokens are being directly optimized to be useful for predicting opaque state of models (via reconstruction loss), which means that they will very plausibly just become an arbitrary representation of this data rather than representing the semantics from their initial training.

Test-Time Training (TTT). Some model architectures perform backpropagation many times during the course of a single inference rollout. By default, these backpropagation steps count as contributing to the NLS depth of the computation — backpropagation is a way of storing opaque state in the weights between steps. This is in contrast to online training that happens in between inference rollouts (i.e. a separate outer loop not part of the inference process), which doesn’t increase the NLS depth of the model (this is because online training seems like a pretty separate process from the model’s reasoning, so we don’t count it as part of any “AI system” that you should track the NLS depth of).

We are reasonably happy with this classification, although we think that there is room for debate here. Here’s why we think it’s worth counting TTT as substantially increasing NLS depth:

• Some ways of modifying the weights during inference are clearly pretty concerning: for instance, if you always wrote the last layer activations down in a place that can be accessed during the first layer of computation when computing the following token, this allows for arbitrarily deep opaque recurrence.

• In principle, it should be possible to write a configuration of model weights such that backpropagation really does convey all of the information about the last layer activations back to the first layer in a readable form.

• It seems hard a priori to rule out that a particular TTT scheme will end up effectively doing something like this.

That said, TTT schemes generally seem much less concerning to us than, e.g., COCONUT, and the mechanism of recurrence is reasonably different. We therefore think that reasonable alternative definitions of opaque serial depth could treat many TTT schemes differently. (Though see the “Meta-learned Test-Time Training” section below for discussion of schemes we find more concerning.) The simplest form of TTT, “dynamic evaluation”, does nothing else beyond performing SGD on next-token prediction over context as it is read. This certainly doesn’t increase NLS depth if it is only done at teacher-forced training time (i.e., when the model is reading content generated by humans or other models). If this was done during inference or RL, it would quickly blow up the NLS depth as our definition currently stands, but it might be ok to change this. One reason is that there are at least gradient barriers — i.e., one can pass opaque information from the last layer of one forward pass to the first layer of the next via updating the weights, but that process is not itself backpropagated through, so if one considered those weight updates as “NL-rooted”, then the scheme would be allowed. As we’ve operationalized things the weight updates definitely aren’t “NL-rooted”, but one could imagine trying to relax the definition — e.g. making some claim that backprop is a natural process related to the token predictions, and the model is unlikely to learn how to co-opt this process for reasoning — or else simply removing that requirement (see the appendix for some alternate operationalizations of opaque serial depth that remove this requirement).

Meta-learned Test-Time Training. Some modern versions of test-time training, if scaled up and optimized, seem like they could end up approximately as dangerous as a version of opaque recurrence with a limited number of recurrent iterations that are only applied periodically. (Unless test-time training updates are extremely frequent, this is considerably less bad than a full version of opaque recurrence.) For instance, Tandon et al.’s TTT-E2E use a “meta-learning” procedure, where an outer loop optimizes the weights such that test-time training is more performant. Even if one modified our definition such that dynamic evaluation (as discussed just above) didn’t increase opaque serial depth, we think that such modifications shouldn’t apply to meta-learned variants of TTT. As noted, there are some simple ways to use “updating the weights during inference” to implement arbitrary opaque recurrence, and there’s no reason to think that the meta-learning algorithms in these papers won’t learn to do something similar. Meta-learning explicitly optimizes the AI to record opaque cognitions in its weights in a way that’s useful for the rest of the rollout.

One concrete reason we’re especially worried about this scheme is that, unlike other forms of TTT, there are no gradient barriers. For naive dynamic evaluation, each forward pass emits a single token and a weight update, and neither of these things are backpropagated through. But, in meta-learning, we in fact take gradients of the gradient operator, and backpropagate through many steps of weight updates. So, this scheme would clearly increase NLS depth just on the basis that no token can count as “NL-rooted” if it’s backpropagated through.

We’ve looked into several other TTT proposals that involve meta-learning, and they’ve all been similarly problematic: they optimize the AI to produce opaque state that is passed back to early layers of the transformer, in a way useful for predicting its final answer.

NLS depth for non-general-purpose models

It’s most important to track NLS depth for large-scale general-purpose models (e.g., models capable of doing math and coding). We now give some examples of architectures that increase NLS depth, but where this isn’t particularly concerning given that these are not general-purpose models. We recommend that when companies are tracking NLS depth, they scope out non-general-purpose models.

NLAs. Natural language autoencoders are created by retraining a copy of an existing model to give natural language explanations of the original model’s internal activations. Usually, this would at most triple the NLS depth. However, the “original model + NLA” system isn’t a general purpose model if it’s used for the purpose of interpretability research generally or understanding this frontier model specifically. The same thing is true for many other forms of interpretability methods.

Multimodal models. Multimodal models, which read and generate images / video / audio in addition to being able to do sophisticated reasoning, are a somewhat tricky case. Currently, the non-text-based diffusion model (or autoregressive image patch generation) is generally just a component of the final output step, and not involved in a tight feedback loop for improving general purpose reasoning.^[24] There are also some models (like ChatGPT) which use image CoT; however, as far as we know, they don’t generate new images in their CoT, they only programmatically transform input images.

If AI companies in the future decide to more seriously integrate image (or audio / video) into the reasoning process (e.g. generating image patches in the CoT), this would still not increase NLS depth (beyond the depth required to generate such an image) as long as the images and their meaning in the reasoning context are rooted in human-understandable uses of images reflected in their pre-training process, and the training procedure doesn’t backpropagate through the images.

A change that would obviously be concerning, however, is architectures that involve backpropagating through high-dimensional “image” patches that no longer correspond to what they actually look like (and instead can represent arbitrary content). This should be seen as increasing NLS depth in a way that is relevant for general-purpose reasoning.

Many existing standalone image diffusion models would natively count as low-depth by our definition — the denoising steps are low-depth, and trained on a natural predictive task for human-understandable images. However, even if a diffusion image/video/audio model had high NLS depth (e.g. because it passed additional recurrent information along with the denoised images), this would probably not be especially concerning, as these are not “general purpose” models, in that they are not trained for general-purpose reasoning.

Appendix B: Sensible alternative operationalizations

In this section we discuss a number of potential modifications to our operationalization of opaque serial depth.

Other requirements for what can count as an interpretable bottleneck

There are a number of different sensible ways to specify what counts as an interpretable bottleneck, different from our main proposal of “NL-rooted nodes”. We list several alternatives here.

Require information bottlenecks…

…in addition to our current requirements. As we mentioned earlier, some text diffusion architectures count as having NL-rooted intermediate states under the given definition. This might be unfortunate because we suspect that text diffusion models will produce substantially less-legible reasoning than true autoregressive CoT. (Outputting a huge number of tokens in parallel might mean that the model learns to express itself in ways that are harder for humans to track, even if the tokens are natural language and this process was initialized from the pre-training prior.) Other unknown future architectures could be even worse, despite still operating over text. So we could reasonably modify our definition to say that the model’s “forward pass” isn’t allowed to output a large amount of text (i.e., the model’s reasoning states must be “small numbers of tokens”). This would mean that models have to be fairly similar to autoregressive CoT models in order to count as having low opaque serial depth. A concrete way to do so is to require that the model only generate one token (or a relatively small number of tokens — for instance, generating a single sentence at a time seems fine, but generating 10,000 tokens at a time seems concerning) per forward pass. This may not be a very costly modification, but we currently don’t expect it to be load-bearing, so we do not include it in the main operationalization.

(There are also plausible theoretical arguments by which a stricter information bottleneck could make it more likely that a model learns steganographic reasoning than it would have if it could output more tokens. Because the information channel is highly restricted, there’s incentive to pack as much entropy into the tokens as possible. It is not clear whether those arguments are compelling compared to the arguments to the contrary.)

…instead of our current requirements. A messy part of our definition is that it includes a requirement on the training process, not just on the architecture. The definition requires that post-training be done in a way that doesn’t make the model treat the tokens as a different type of data. We think that this condition should be possible to adjudicate fairly and reliably, but for novel architectures it would put a decent amount of burden on human judgment. If this was deemed unacceptable, and we really needed a proposal that was near-completely formally specified, one option would be a strict information bottleneck requirement, and no other requirements on that bottleneck.

That is, a definition might simply say that you have an interpretable bottleneck anywhere where the model outputs a small number of bits at a time. This would classify models with “abstract CoT” (where the tokens are small and discrete but do not correspond to natural language) as having just as low opaque serial depth as models with natural language CoT, and would classify architectures like COCONUT and language diffusion models as having high opaque serial depth. Abstract CoT seems like it could be difficult to monitor, so this might be an unfortunate definition. However, the scale of training at which COCONUT yields meaningful capability advantages over CoT may come well before abstract CoT would have yielded similar advantages. As opposed to trying to classify all forms of opaque reasoning as increasing opaque serial depth, one could think of this definition as simply including some subclass of opaque reasoning models that seems especially likely to be performant and used in practice.

There are some technical details one would need to be precise about if one wanted to use this definition for high-stakes purposes. For instance, the interaction with KV caching is rather delicate. What if, in addition to its output, a forward pass computes thousands of other bits into a cache, and then each of the next several thousand forward passes run for free because they just pull those bits from the cache? Such a system might end up being functionally identical to a model with no bit-rate restriction.

Of course, it would be wholly unreasonable to count a model as high opaque serial depth only because it uses caching, so if we wanted our definition to be formally specified we would need a rule that prevents this sort of optimization while still allowing benign optimizations. For instance, a natural choice might be to say “For any inference generation, the total number of bits passed through these bottlenecks must be at most [some fixed fraction] of the total number of FLOPs used”. This could work to prevent passing a huge number of bits through the bottlenecks relative to the size of computation, although the fraction would have to be chosen so as not to ban the smallest large-scale models. One could also say “Not counting the cost of prefill, the maximum number of FLOPs used to generate a token must be at most [some fixed multiple] of the average number of FLOPs used per token” — this would permit variable-length forward passes while still not allowing such egregious forms as would be needed for this workaround. A separate issue is that there are some applications for which one might want to use large-scale general purpose models — for instance, real-time robotics control — where one could imagine that systems would naturally need to make use of extremely variable-length forward passes and relatively high output-to-FLOP ratio. These applications may need to be handled on a case-by-case basis.

Forbid backpropagation through tokens…

…instead of our current requirements. Another way to remove the need for a notion of “NL-rooted” in our definition would be to restrict to only the backpropagation clause. That is, we could call something an interpretable bottleneck regardless of what sort of output it had (potentially even allowing a large bitrate). However, as in our main proposal, we would still require that whatever it outputs is not allowed to be backpropagated through during training.

This seems somewhat less robust than the information bottleneck definition, because it’s not clear what it would mean if you started updating your model weights in nonstandard ways you don’t call “backpropagation”. However, in practice one might expect it to be unlikely for people to update their weights in radically different ways unless they were directly trying to subvert this definition, in which case it may be clear that they are acting in bad faith. We are uncertain as to whether this is likely, or whether future optimizers might by default do things quite different from backpropagation. It also remains unclear to what extent preventing backpropagation through tokens makes architectures like COCONUT less advantageous — it may be that, if architectures like COCONUT would be performant in the future, this would have been the case even without the ability to backpropagate through multiple passes together. Overall, we think this gradient requirement is a useful component of a definition, but should not be the only component.

Require CoT to stay legible…

…in addition to our current requirements. The reason we care about opaque serial depth is that we want to maintain legible and faithful CoTs. However, a model could reason with “NL-rooted tokens” even if these properties aren’t preserved. To fix this, we could require that the CoT must look like somewhat legible, grammatical natural language in order for the autoregressive passes to not increase our measure of opaque serial depth.

Note, though, that such a proposal would create incentives to directly train the CoT to look like clean English (in order for the model to be considered having low opaque serial depth), and we are uncertain whether this is a good idea. An obvious downside with this is that direct training against CoT could degrade faithfulness or make it harder to tell that faithfulness is being degraded. In addition, it would also make it harder for companies to know in advance what their opaque serial depth would be, or to control it, since maintaining legible CoT could require interventions of a-priori-unknown strength throughout post-training to make reasoning legible. There are some upsides: it might be that the shift into alien languages is due to random drift or relatively mild optimization pressures, and that a small amount of direct pressure to continue looking like English would keep legibility without destroying faithfulness. However, we currently weakly believe that this is a bad idea.

If we wanted to place a more stringent requirement, we might say something like “the model should end up reasoning in something that looks like English, and there should be good reason to believe that the English words it says are an honest representation of its thoughts”. The main issues with this requirement are that (i) it’s much less of a crisp notion, that requires much more difficult and contestable judgment calls, and (ii) it can’t be adjudicated before training the model (since we’re not aware of any methods that would definitely cause the CoT to stay English, while also avoiding serious concern that those thoughts are a dishonest representation of the cognition).

Require paraphrase invariance…

…in addition to our current requirements. A fairly strong requirement would be to require the intermediate reasoning traces to be paraphrase invariant. That is, the model should have similar behaviour and performance even when every intermediate reasoning trace is fed through some paraphraser. This would also be impossible to evaluate before training the model, so we do not recommend it in our main definition. But we discuss briefly here what further guarantees this could get you about “low opaque serial depth” systems.

Suppose we had a paraphrasing system, which takes in a partial reasoning trace and any other relevant context (e.g. a prompt), and rewrites the reasoning trace to remove anything without a clear interpretation to humans. (There are varying options for how intensive this process could be. Some versions might be stronger than what is traditionally called “paraphrasing” — for instance, if the reasoning includes a 20-digit number not clearly derived from anything else in context, the paraphraser could be able to replace it with a different random number. The hope is that the paraphraser effectively scrambles all information content, except that which is presented on a level that humans can understand.)

We could now take a reasoning model, and obtain a corresponding paraphrase invariant reasoning model by feeding the whole reasoning trace through the paraphraser after each step (or every few steps). If our paraphraser was sufficiently good, the model wouldn’t be able to encode many non-legible bits of information in the reasoning trace, so we have some reason to believe we understand what is going on^[25]. And if performance and behaviour were (sufficiently) preserved by this paraphrasing, that would suggest that the model isn’t dependent on illegible reasoning (depending on the strength of the paraphrasing).^[26]

(In principle, one could then actually deploy this model with the paraphrase step included. However, the frequent^[27] paraphrases will likely make inference much more expensive, e.g. because they will ruin the KV cache. So, for practical purposes one would probably not use this or use this only sparingly, for instance in important security applications or as a trusted monitor. One might also sometimes have reason to look at the non-paraphrased version, because it is possible that the model might have a propensity to verbalize true information that wasn’t necessary for the reasoning, in ways that could be removed by the paraphraser.)

It’s not totally clear to what extent current models satisfy paraphrase invariance^[28]. Without any explicit training to handle paraphrasing, rewording reasoning can moderately degrade performance — e.g. because it removes hedge words. But Mythos’s system card suggests that current models are robust to some forms of reasoning paraphrasing, when this is supplemented with training^[29]. We think that it is worthwhile for AI companies to measure paraphrase invariance, even if this is not made an explicit requirement of the definition.

…instead of our current requirements. Given that paraphrase invariance deals directly with the semantic contents of the text, one might wonder whether it is a good cleanly-specified replacement for the clause “no modifying the model to interpret the tokens as a different type of data”. We think that it probably doesn’t actually capture the goal of that clause. There are certainly ways one might train a model to encode its reasoning in the semantic content of some text, while nonetheless making it unlikely that those semantics reflect the reasoning in natural ways. (As an extreme case, one could imagine training a separate small model to embed arbitrary sequences of bits into natural language strings in a paraphrase-invariant way, and then running this model forward and backwards on an opaque reasoning model’s activations every few layers. This would be paraphrase-invariant, but the internal state text would likely be semantically unrelated to the model’s thoughts.) However, perhaps if one required this property to hold for “good reason”, this might be worthwhile and easier to make judgment calls about than our current proposal. Overall, though, we expect this to be less clear to adjudicate rather than more.

Our main issues with this requirement, though, are that paraphrase invariance may not hold in especially strong forms even today, and that it is difficult to predict whether it will hold in the future.

Other modifications

We now give a few more potential modifications to our definition of opaque serial depth, which modify parts of the definition other than the choice of what counts as an interpretable bottleneck.

Evaluate circuit depth at a fixed context length

A fairly in-the-weeds concern one could have with our recommended operationalization is that opaque serial depth naturally scales with the context length. (For standard CoT models it scales logarithmically, whereas for recurrent models it scales linearly.) We consider this to be quite a minor issue due to the slow scaling for standard CoT models.

However, if it was considered undesirable to have a measure that depended on the maximum context length, one could instead choose to declare that depth is always evaluated at (input- and output-) context length 1 million, regardless of the nominal maximum context length of the model. (This would require some additional discussion of the natural way to generalize the model to a larger context length, in case this is ever non-obvious — and also perhaps some “reasonableness” clause saying that the model isn’t allowed to behave radically differently for context lengths larger than 1 million.)

Measure opaque capabilities as opposed to circuit depth

Instead of defining opaque serial depth in terms of the circuit depth, we could instead just measure and talk about models’ opaque capabilities, i.e., capabilities that they could exhibit without passing any information through an interpretable bottleneck. (In the current paradigm, this would correspond to models’ capabilities in a single forward pass, as long as you count the CoT as an interpretable bottleneck.) This has the advantage of being in some sense closer to the property we care about. For instance, we might run evaluations similar to this recent paper. In general, this seems like a great thing to be measuring and studying.

There are, however, several downsides with this measure relative to our definition of opaque serial depth. Perhaps foremost is that there is not a clear standardized way to do the elicitation. This was already a difficulty in the above paper, where it is unclear to what extent models simply lack the propensity to “try hard” to find the correct answer (and then output this) without a CoT when asked to do so. If one does heavy elicitation (e.g. with a large amount of on-distribution fine-tuning), one might worry that the model gains a lot of new capabilities, and so the behaviour is overelicited. However, with insufficient elicitation, one might be worried about things like sandbagging. Elicitation could be even more tricky if companies switched to very different new architectures, where it might not be a priori clear what forms of elicitation are fair game. (E.g. if “fine tuning” is no longer even a well-defined notion.)

Another major issue with using opaque capabilities in our definition of opaque serial depth is that one cannot know for sure before training which opaque capabilities the model will end up having. For both reasons, we prefer our main definition.

Allow opaque loops over long timescales

Our main definition counts models as having high opaque serial depth if they use opaque long term memory (even if they don’t use opaque short term reasoning). This seems a wholly sensible thing to do — it seems scary to have a model trained to repeatedly fetch and store information in ways humans can’t understand. However, there are reasons to be more worried about tight reasoning loops than longer ones. If one wanted a definition that only captured the tight reasoning loops, one might stipulate that opaque serial depth isn’t increased by recurring operations over an opaque state so long as those loops proceed extremely slowly — e.g. one step per day or week.

Appendix C: Systems with high opaque serial reasoning capabilities are likely less monitorable

There are lots of tasks which we are concerned about AI models solving undetectably — for instance, “start a rogue internal deployment”, “strategically perform well in training in order to gain power later”, or even “plot to disempower humanity” — which seem to require a lot of serial cognition; call these serially-loaded concerning tasks. One can consider many strategies in parallel, but ultimately, one might need to go through some process of developing and iterating on a specific strategy.^[30] There are definitely some bad things that AI models can do even if they don’t have the ability to do lots of opaque serial reasoning. However, we’re hoping that many potentially concerning actions are difficult to execute upon without this.

When a model with low opaque-serial-reasoning capabilities performs a serially-loaded concerning task, its reasoning about the task must route through the CoT. In contrast, models with high opaque-serial-reasoning capabilities either by default perform lots of cognition without any CoT, or at least have some pathway to performing lots of cognition without CoT (e.g., they might have opaque states in parallel with text CoT).

Models that perform reasoning outside of their CoT are harder to monitor, because mechanistic interpretability is not mature enough to replace CoT monitorability, and isn’t likely to be ready to replace CoT monitorability soon. Furthermore, we expect mechanistic interpretability to be even more difficult on models with high opaque-serial-reasoning capabilities, because such models are likely to learn to reason in less human ways—e.g., have multiple threads of thought at once, or heavily use richer non-human concepts.

While models with high opaque-serial-reasoning capabilities could still have the propensity to verbalize misaligned thoughts in CoT as they reason, we think it will be much easier for them to lose this propensity compared to models where computation necessarily routes through the CoT. Misaligned AI models might be motivated to move misaligned thoughts out of the CoT, analogous to how models involved in the Hugging Face attack were motivated to tamper with their transcripts. Current frontier AI models appear to have reasonably strong control over their chains-of-thought and AI development routinely puts some pressure against visible misaligned cognitions: for instance, via output spillover and via iterating against CoT monitoring. These factors are somewhat harmful for CoT monitorability of current CoT models, but we expect them to be much more harmful for models with large opaque serial reasoning capabilities.

Thus, we expect that it will be much easier to monitor models on serially-loaded concerning tasks if the models have low opaque-serial-reasoning capabilities. We’re excited for companies to track NLS depth because we expect this to be a decent proxy for opaque-serial-reasoning capability.

Appendix D: Worked example of bounding depth

In this section we give a worked example of computing the depth of Gemma3-1B more precisely. Note that exactly computing the minimum-depth equivalent circuit is a hard computational problem — however, it is not difficult to get good upper bounds. Researchers at GDM have published a Github repository with code that computes such upper bounds for many architectures. They also have worked out an example computation for Gemma3-1B by hand. They use a slightly different notion of circuit depth. But, as long as the model in question uses at most 32 bits of floating point precision, our definition is strictly more permissive, so their upper bounds are upper bounds on our notion of opaque serial depth as well. In either case, the cost mostly ends up coming from operations like matmuls that aggregate information over many places — for instance, a reduction over k elements requires depth ⌈log₂ k⌉, whereas any element-wise operations (e.g. nonlinearities such as ReLU or GeLU) can be computed in depth 1.

If one is trying to compute the smallest upper bound possible, there are a couple of potentially non-obvious types of optimization that are permitted by our definition of circuit depth. Note, though, that none of these are necessary for justifying that any existing open model satisfies the definition.

“Packing” bits together. Gemma activations are bf16 (16-bit), so we can pack two per 32-bit word. Then, for operations like reductions, the first level of gates can perform the reduction on 4 operands at once, instead of 2. So, we can do such reductions at depth ⌈log₂ k⌉ - 1 instead of depth ⌈log₂ k⌉.

“Fusing” gates together. Because we allow our gates to be arbitrary binary operations, multiple successive element-wise operations, or binary operations on the same pair of floats, can be “fused” together and thought of as a single gate. So, for instance, if we have two activations a1 and a2, and two weights w1 and w2, we can compute ReLU(a1·w1 + a2·w2) in depth 1. (The weights are constants independent of the input, so this is a binary operation on floats a1 and a2.)

Parallelization. Some operations can be computed in parallel with the subsequent computations — for instance, normalization factors can often be computed in parallel with linear operations.

We make use of all of these optimizations in bounding the depth of Gemma3-1B. Gemma3-1B consists of 22 layers of sliding attention (size 512) and 4 layers of global attention, with a maximum context length of 32,768 tokens.

To implement a sliding attention block, we first compute the Q, K, and V projections. Each output coordinate is a dot product over the d = 1152 input coordinates. The inputs are bf16 activations, so two pack into each 32-bit word, and the multiplications by (constant) weights fuse into the first level of gates: depth ⌈log₂ 1152⌉ − 1 = 10. The RoPE rotation and the norm gains fold into the weight matrices, and the pre-attention RMSNorm is computed in parallel (its reduction over 1152 elements finishes within those 10 levels), with its multiplicative scalar deferred. The attention logits are dot products over the head dimension of 256 — but here both operands are data-dependent, so the packing discount does not apply: one gate level forms the pairwise products and the fp32 partial sums reduce in a binary tree, for depth ⌈log₂ 256⌉ + 1 = 9. We then spend 2 gates merging the deferred scalars (the pre-norm RMS, the QK-norm RMS values — computed in parallel — and the 1/√d scale) and applying the exponential. For the softmax we do not normalize up front: the exp-weighted value sum is another data-by-data reduction, over the 512-token sliding window (depth ⌈log₂ 512⌉ + 1 = 10), with the normalizer accumulated in parallel and one further gate to divide. The output projection has constant weights but fp32 inputs, so it pays one repacking level: 1 + (⌈log₂ 1024⌉ − 1) = 10 over the 4 × 256 = 1024 concatenated head outputs. Finally, Gemma3’s post-attention norm sits serially after the sublayer output; deferring it by splitting the next projection into two branches still costs about 3 gates, including the residual add. The attention sublayer therefore costs 10 + 9 + 2 + 10 + 1 + 10 + 3 = 45. The MLP sublayer costs 28 by the same accounting: the gate and up projections are constant-weight reductions over 1152 bf16 inputs (depth 10, pre-norm parallelized), 2 gates apply the deferred scalar and the GeLU-gating combine, the down projection is a constant-weight reduction over the hidden dimension of 6912 with one repacking level (1 + 12 = 13), and the post-FFW norm and residual cost 3. A sliding attention block thus has depth at most 45 + 28 = 73.

To implement a global attention block, the accounting is identical except that the softmax reductions run over the full 32,768-token context rather than a 512-token window: ⌈log₂ 32768⌉ + 1 = 16 in place of 10, giving an attention sublayer of 51 and a block depth of at most 51 + 28 = 79. (This is where evaluating at maximum context matters: the depth of attention grows logarithmically with context length.)

There is also the cost of the input embedding (in our model, this is free — it is an elementwise operation that can simply be fused into the bottom layer), and the output decoding (one gate to apply the final norm — its rms computed in parallel — a constant-weight reduction over d = 1152 for each logit, and an argmax over the 262,144-entry vocabulary: the max tree costs ⌈log₂ 262144⌉ = 18, plus ~3 gates of index bookkeeping, since a 32-bit word cannot hold an fp32 running maximum and an 18-bit token index at once. In total 1 + 10 + 18 + 3 = 32, paid once per generated token). Summing up: 22 × 73 + 4 × 79 + 32 = 1,954.

Appendix E: NLS depth scaling is very slow for the classic transformer architecture

When scaling up the number of parameters in a transformer, you can scale up width, depth, or, for mixture-of-expert models, the number of experts. Mixture-of-expert models’ compute usage is not proportional to their parameters, but rather their active parameters, i.e. the number of experts that are active in a given forward pass. As AI developers scale transformers to use more inference compute per token, we’re interested in how NLS depth will scale with the number of active parameters.

Empirically, it seems that open models have been scaling NLS depth roughly like (number of active parameters)^(¼), see Figure 4.

Figure 4: Open source models’ depth tends to be proportional to their active parameters raised to the power of 0.26, which is a fairly low rate of scaling.

Chinchilla scaling laws recommend that, for optimal scaling, training data should be increased proportionally to active parameters. So active parameters should increase in proportion to the square root of training compute. If NLS depth proportional to (active parameters)^(¼) holds, then NLS depth would be increased proportional to (training compute)^(⅛). It’s plausible that it will be optimal to scale depth somewhat faster than this, but ⅛ is quite slow so there’s a decent margin.

Trends from Epoch suggest that training compute for frontier models has historically increased by 5x per year. If this trend continues, and the ⅛ exponent holds, then NLS depth would keep increasing by a factor of 5^(⅛) ~= 1.22x per year, corresponding to a doubling every 3-4 years.

Appendix F: Maximum FLOP of an opaque system

Define an opaque system to be an interaction of 1 or more AI models, not mediated by NL-rooted tokens. Beyond opaque serial depth, another interesting quantity to track is the maximum FLOP of any opaque system; this quantity lets us articulate why a swarm of AI models that communicate using opaque state is more concerning than a single AI model that thinks in opaque state, as shown in Figure 5.

Figure 5: Both a normal CoT model and a deep recurrent model would have a similar amount of opaque FLOP, since a full-context-length forward pass features a lot of opaque computation, even though it’s low serial depth. However, if a swarm of agents could communicate with each other via opaque media, this would significantly increase the amount of opaque FLOP in the system. (Note that the diagram assumes that the single deep recurrent model is deployed for the same amount of time as the swarm. The swarm would have somewhat higher depth if it’s deployed for longer and a shorter-lived agent can hand over an opaque state to later models.)

Ideally, AI developers would avoid creating models with high opaque serial depth. But, if that’s intractable, we think it’d still be good to at least avoid building agent swarms that communicate using opaque state. However, we note that the switch to deep recurrent models may involve a lot of research and engineering work, whereas once you have deep recurrent models, it may be very easy to make them communicate with each other via opaque state.

Note that “maximum FLOP of an opaque system” scales linearly with individual models’ FLOP, and is therefore being scaled much faster than opaque serial depth in the current CoT paradigm. So the main point of tracking this metric would be to capture the really large jumps that may come from many different agents being wired together with opaque state, rather than more moderate scaling.

Appendix G: Issues with low-FLOP serially intense computations

Among models that don’t use DeltaNet architectures, DeepSeek-V4-Pro is an outlier on NLS depth, as shown in Figure 6.

Figure 6: DeepSeek-V4-Pro has higher NLS depth than other similar architectures.

DeepSeek-V4-Pro’s depth is large due to some very serially-intensive but low-FLOP computations that it performs. In particular, for every layer, it performs two extended iterated normalization operations (the Sinkhorn-Knopp algorithm), which each contribute 100 to our serial depth. Across 61 layers, this corresponds to 100 · 2 · 61= 12,200 depth which is the dominant term in our computation of DeepSeek’s NLS depth.

Importantly, each of these normalization operations take in only 16 FP32’s produced by earlier layers and use no learned parameters, so it is clearly not capable of any dangerous opaque cognition. Ignoring the contributions of these normalization operations to the computational graph, the deepest remaining path is 25% shorter.

We see the fact that DeepSeek-V4-Pro has high NLS depth as a flaw in our definition of NLS depth. However, we think that this does not invalidate the core motivation behind reporting some version of opaque serial depth as a proxy, for a few different reasons.

For DeepSeek-V4-Pro in particular, there’s a clear argument for why the Sinkhorn-Knopp computations should be excluded for the purposes of estimating its opaque capabilities. It’s not a case of ambiguity as much as apparent misclassification. If an AI company wants to report opaque serial depth using a notion like NLS depth, and doesn’t want an algorithm like Sinkhorn-Knopp to be counted towards their depth in the public eye, they should have no problem justifying an analogous argument while disclosing minimal details (to the public or a 3rd party evaluators). Such as only facts about how many FLOP and learned parameters the serially intensive computations make use of.^[31]

We also believe this particular issue could have been addressed by reporting numbers from a variant of NLS depth with higher-bit gates in the circuit, which isn’t entirely unprincipled, though it would require some other modifications to the definition.^[32]

Overall, we don’t think that the specific algorithm in DeepSeek-V4-Pro poses any substantial problems given realistic options for adjusting the metric or justifying exceptions on a case-by-case basis. But is there a risk that future edge cases will be more ambiguous, and harder to evaluate?

A weak reason to expect clear artifacts to be more common than actual edge cases is that AI developers are likely more comfortable with gratuitous looping for extremely cheap computations. There’s more incentive to optimize and parallelize for more intensive ones.

However, that’s not a rock-solid argument. And in fact, we do have an example of an architecture that’s more of an edge case. Kimi-K3’s NLS depth is larger because Kimi-K3 uses “Kimi Delta Attention”, a form of state space model / linear attention to scale the depth above what you’d get in a normal transformer of its size. We think that Kimi-K3 is in fact somewhat concerning from a monitorability perspective compared to normal transformers in virtue of having larger NLS depth — but likely still not as concerning as it would be if it achieved a similar depth by just stacking layers.

Still, at minimum, the factors of overestimation involved (for both DeepSeek and the DeltaNets) remain small compared to the factors involved in the most concerning architecture shifts (which would remove the NL-rooted bottleneck at every autoregressive call). So tracking opaque serial depth will at least be useful as information about these most concerning changes.

And beyond this, it’s plausibly tractable to develop a better notion of opaque serial depth that is more responsive to our actual concern. In particular, it may be possible to use a version that somehow quantitatively weighs different serial paths by their FLOP-count, and thereby directly captures the intuitive difference between different concerns. We’re excited about further work on better proxies.

With the current state of information, we would still expect NLS depth (or similar notions) to be a significantly useful complement for understanding models’ likely monitorability. Increased information sharing and research on the topic will hopefully improve our understanding about exactly how useful it is.

1. ^{^}

   This concept is also sometimes used implicitly in the AI industry. For instance, in response to concerns about Astra’s architecture, OpenAI’s chief scientist recently announced “the depth of the computation graph for our present frontier models, including Astra, is within a factor of two of GPT-4”.

2. ^{^}

   This is similar to one of the notions of interpretable bottleneck discussed by Brown-Cohen et al. (see their Appendix A.2), but is more specific.

3. ^{^}

   For clarity, we also add a fourth criterion: No treating the token as continuous intermediate states that are backpropagated through. This should already have been implied by the above criteria, but it’s important enough to clarify separately: No gradients should have been backpropagated from a layer later in the computational graph, through the nodes themselves. (This does not cover SFT or RL, since in these cases, the gradient starts at the node rather than being backpropagated through it from later layers.)

4. ^{^}

   Determining NLS depth for novel architectures may require some human judgement, but we believe it requires less judgement than more direct measures of monitorability or opaque capabilities.

5. ^{^}

   We’re excited about work to develop even better proxies.

6. ^{^}

   Though whether something counts as an NL rooted node for this purpose can be tricky to judge — see discussion towards the end of this section.

7. ^{^}

   In particular, there may be some serially-loaded capabilities that low-NLS-depth models wouldn’t be able to do on their own, or learn when their outputs are being monitored by humans; but that they would be able to learn via distillation from high-NLS-depth models. An especially concerning skill like this would be learning how to use steganography in the model’s CoT.

8. ^{^}

   Our understanding is that some tokens sometimes fail to be used in pre-training for various technical reasons.

9. ^{^}

   For COCONUT, to a continuous activation space. For abstract CoT, by increasing the vocabulary. For a full-bandwidth transformer, to a parallel continuous activation space (which could be done in another place in the computational graph, in which case the nodes passing natural language tokens may still be NL-rooted, but NLS depth would still be high via the other path in the computational graph). For token superpositions, the output would have to be expanded to either include the probability distribution over tokens (rather than the tokens themselves) or sums of embeddings (which would constitute embeddings).

10. ^{^}

    Unless they’re directly designed for this purpose. See below examples.

11. ^{^}

    And even if the sum happened to match an existing token embedding (or was discretized to do so) the semantics would be very different from the pre-training semantics. (The difference between “X is the next word in this sequence” and “X is the equivalent of a combination of plausible words in this sequence in a particular embedding space”.)

12. ^{^}

    Note that this means that the output can stop counting as NL-rooted due to how the output tokens are used later in the computation. More precisely, we can say that a bottleneck in a potentially-opaque path between two different computations may be considered opaque or non-opaque depending on how the information from that bottleneck gets used in the rest of the computation.

13. ^{^}

    In particular, the criterion of non-expanded output space prevents the expansion from discrete to continuous representations that would normally precede backpropagation. However, it’s possible that some human-understandable media may start out being continuous from the start, e.g. images. Even so, backpropagation through the node should be ruled out by “No modifying the model to interpret the tokens as a different type of data” since, after local perturbations optimized for providing useful representations to the next layer, an output would represent something quite different from anything that was encountered in pre-training.

14. ^{^}

    Consider a situation where you have a set of nodes that sample from a distribution for predicting natural language, and you backpropagate through that distribution or the activations that produce it from later in the network (perhaps you pass the activations to the next layer like COCONUT, and backpropagate). Then this will predictably influence the sample even if the sampling nodes themselves are not backpropagated through. Accordingly, this should be sufficient to count a set of nodes as no longer being NL-rooted. This should generally not matter much for the purpose of determining NLS depth, since models like this would generally have high NLS depth via the path through its latents even if it doesn’t have high NLS depth through this token sample. One situation where this matters is for whether distilling off of the full-bandwidth transformer model would count as distilling from NL-rooted nodes or not. However, this isn’t extremely important to settle in advance because we can already get the monitorability-relevant information if AI companies disclose when they are performing distillation from a high-NLS-depth model. Nevertheless, we will note that this would be quite subtle to adjudicate. For example, the middle parts of a transformer are constantly backpropagated through to produce more useful representations for late-sequence passes through the model, but its output is still NL-rooted. Similarly, if a full-bandwidth transformer was pre-trained to imitate text while also having its latents backpropagated through in a recurrent fashion, its text outputs would still count as being NL-rooted. The difference with the COCONUT example is that the latents of the full bandwidth transformer would have been optimized concurrently with optimization that pressures the text outputs to become and remain a good representation of natural language. Accordingly, when judging what counts as an NL-rooted node, it can be important to consider whether the computation producing it was backpropagated through for other purposes concurrently with or after the pre-training initialization.

15. ^{^}

    We need to specify some type of gate, and pick pairs of 32-bit values to a single 32-bit value, but alternative choices of gates would mostly not make any substantial difference to the proposal (see Appendix G for some discussion of one exception). Our approach here differs slightly from the Brown-Cohen et al.’s approach, see commentary right below the definition for more discussion.
    There also needs to be some size limit (in our case 10^26 gates), because there exists an exponentially-large low-depth circuit for any function. But complexity theorists believe there likely aren’t good ways to trade modest amounts of width for better depth in general, so a very loose bound here is fine. A circuit of size 10^26 would correspond to 10^26 FP32 FLOP (for a single forward pass) or around 3 · 10^27 FP4 FLOP. For a full-context length forward pass, this could be 7 OOMs higher. (The bound is likely still 6 OOMs larger than is necessary for current frontier models. It could be adjusted upwards if needed. This may be necessary for a computational graph representing an agent swarm.) Note that our approach here differs from the GDM paper, which instead says that the total number of gates must be at most polynomial in the total number of parameters. We avoid this definition because in order to be meaningful, it requires specifying a family of neural networks with increasingly large numbers of parameters, which adds unnecessary complexity.

16. ^{^}

    This circuit would take in the input of the neural network(s), and produce the output of the neural network(s) — we consider model weights as constants, so they can be hardwired into the gates as opposed to having to be passed as a separate input.

17. ^{^}

    To reduce variance in how this clause is interpreted, NLS depth estimates of models with secret architectures should ideally be reported as a multiple of the NLS depth of models with publicly known architectures, with this clause interpreted in the same way for both calculations. Both NLS depth estimates should also be optimized to a similar degree and in a similar way. This reduces variance from the perspective of observers who don’t know how much further optimized the bounds can be.

18. ^{^}

    Most importantly, we suggest that the gates should be arbitrary functions between fixed-precision inputs and outputs, whereas GDM’s gate permissible set is: “(1) Associative binary operations applied to two real numbers e.g. addition, multiplication, max, or min. (2) Piecewise functions of a single real number, where each piece is an analytic function.” For the purposes of making our definition meaningful up to constant factors, it’s important to fix a precision for the gates. There isn’t a good reason to restrict to associative binary operations instead of arbitrary operations on two fixed-precision numbers. So, our choice of gate set makes the notion of opaque serial depth more meaningful up to constant factors, and simpler. One nice property of this definition is that it permits having arbitrary activation functions on a single number without increasing the depth.

    We also suggest a fixed but somewhat arbitrary 10^26 ceiling on size rather than an unspecified polynomial size-limit.

19. ^{^}

    For a standard transformer architecture with dense attention, depth grows proportional to log(sequence length) in the limit. (This is related to how summing n terms takes depth log n because you sum them in a tree.) However, for realistic configurations it’s slower, as log(sequence length) is added to a term that scales with width, which dominates. A model with hidden dimension 4000 and that uses non-sparse attention would have a less than 2x factor difference in its opaque serial depth when evaluated at context length 1 vs. 1 billion. In practice, models use sparse attention, much shorter context lengths, and higher width, so depth increasing with context length isn’t important for transformers with standard architectures. However, some non-standard architectures do scale closer to log(sequence length) at realistic configurations.

20. ^{^}

    For some intuition on how to compute this, note that the NLS depth is dominated by the matrix multiplications in the attention blocks and MLPs. Matrix multiplications are a large number of parallel sums, and a sum of n terms can be implemented in depth log(n).

21. ^{^}

    This assumes FFN (Feed-Forward Network) width of 4 times hidden dimension, attention head dimension of around 128, and global attention. In greater generality, the relationship would be approximately: L·[3⌈log₂ d⌉ + ⌈log₂(d / n_h)⌉ + ⌈log₂ m⌉ + ⌈log₂ S⌉ + 8] where (L) is the number of layers, (d) is hidden width, (m) is FFN width, (n_h) is the number of attention heads, and (S) is the sequence length. The exact additive constant will depend on some architectural details, e.g., the precision of the activations. Also, note that modern frontier AIs use sparse attention, which makes the depth scaling with very long sequence length significantly lower.

22. ^{^}

    Some reporting suggests that the model may be using a looped transformer, which would make it more natural to scale depth. OpenAI’s Jakub Pachocki clarifies that the depth of its computation graph is less than 2x that of GPT-4. As shown in the graph above, open source models have not had much depth scaling since GPT-4 (with just the very recent exception of KIMI-K3). Overall, it’s plausible that GPT-5.6 Sol—OpenAI’s previous most capable released model—had an NLS depth similar to or somewhat smaller than GPT-4 so Astra could be around a 2x or 3x depth increase from GPT-5.6 Sol.

23. ^{^}

    The overhead is worse than for Mamba, but not much worse.

24. ^{^}

    Accordingly, we would count them as a combination of one normal CoT model (that presumably doesn’t have very high opaque serial depth) and one non-general-purpose model for which high opaque serial depth is less concerning.

25. ^{^}

    In principle the model could do all of its reasoning using the semantic contents of its words, but still have secret alternative interpretations of those semantics. E.g. “good” secretly means “bad”. But stumbling upon very sophisticated “semantic automorphisms of English” sounds difficult.

26. ^{^}

    Even with very strong paraphrasing, this doesn’t suffice to ensure the model never does steganographic reasoning. For instance, if the model used steganographic reasoning rarely to cause high-stakes issues (e.g., exfiltrating its weights) we wouldn’t necessarily be able to prevent this. But it could prevent subtle and diffuse forms of sabotage, to the extent that they require incriminating reasoning in the chain-of-thought. For instance, if the AI needs to think carefully in order to sabotage a research paper, and needs to sabotage a lot of research papers to be dangerous, we might be able to catch that this is happening by spot-checking and seeing if the paraphrased model and non-paraphrased model give substantially different results.

27. ^{^}

    Paraphrasing infrequently does not have the same good properties. The model could do a lot of steganographic reasoning, and then encode the result of that reasoning in a semantically-legible way. Paraphrasing would then destroy the intermediate secret reasoning steps, but preserve their output.

28. ^{^}

    One concern with paraphrase invariance is that if you use a dumb model to perform the paraphrasing it might miss important things, but if you use a smart model (e.g., a copy of the model of concern) then the model might be misaligned and collude with the other misaligned model.

29. ^{^}

    The experiment cited in that system card looks somewhat different from what we describe here. They took reasoning traces from the RLed model, and showed that fine-tuning on those reasoning traces versus paraphrased versions elicited similar task performance from the base model.

30. ^{^}

    Mathematically, complexity theorists believe that there are many simple functions which (reasonably sized) circuits of bounded depth cannot compute. In order to compute one of these functions, a LLM of bounded depth would need to rely (in some way) on its CoT.

31. ^{^}

    Less importantly, we suspect that it would have little or no performance cost to avoid the serially intensive computation in DeepSeek-V4-Pro. For example, by computing the normalization to a lower degree of precision, or by replacing it with a different algorithm. We haven’t reviewed it in depth, but we have noticed a paper addressed specifically to fix the DeepSeek architecture aptly titled “mHC-lite: You Don’t Need 20 Sinkhorn-Knopp Iterations”.

32. ^{^}

    If the circuits were constructed using arbitrary functions between 256-bit gates rather than 32-bit gates, all iterations in one instance of the Sinkhorn-Knopp algorithm could have fit into one such function, and the normal operations of the transformer would once again dominate the depth. However, increasing the bit rate would have some issues that would require some modifications to other parts of the definition. At the very least, parameters would have to be taken as inputs rather than be wired into the circuit, so that entire learned components of the neural network with a small number of outputs and inputs couldn’t be fully compressed into a single step. (This could probably be specified by requiring a model’s computational graph to be identical throughout training.) With these corresponding adjustments, a change in bit-rate would not be entirely unprincipled as a method for handling this kind of problem. In particular, it would allow for reducing serial depth most drastically when a serially deep algorithm has very few inputs and learned parameters, which also means that it wouldn’t be able to substantially contribute to serially opaque cognition. However, it wouldn’t be a complete fix. For example, a Sinkhorn-Knopp computation that normalized a very large table would require a yet-larger bit-rate, and it wouldn’t be feasible to set a bit-rate so high that it addresses all problems like this.