Princeton Ouro-RLTT 分支循环 Transformer：评估器探针与分支选择研究

Evaluator probes and branch-selection scaffolding for reading hidden-state signals in Princeton Ouro-RLTT loop states.

Current status (paper v3, 2026-07-27): the readout--control boundary has been located rather than merely observed. Five sealed, pre-registered conversions asked where a frozen readout turns into an outcome gain. Decision-level use converts: calibrated abstention improves the risk--coverage trade-off in all four sealed arms across both domains, and forced terminal selection clears an exact matched-random null even on a pool where every candidate is well-formed by construction (27/32 vs 64.8% expected, exact p = 0.0086) -- which resolves the content-sensitivity question the v2 pool could not. Generative control does not: directional steering is an established negative, the four-task fork screen is bounded, per-task recurrent-depth allocation is a sealed null (neither the tap nor the model's own exit gate beats a matched-histogram random control, on both checkpoints), and a minimal sign-conditioned LoRA returns a bounded no-detected-gain result. Phase 2a branch survival remains locked under DualAnchor, and the branch-carry substrate and bit-exact splice remain validated. Full training-time integration is still the next lever and has not been run.

⚠️ Correction notice (2026-07-25)

Several figures quoted in this README were corrected by a project-wide evaluation audit (source-item leakage across row-level splits; a presentation-order prior in the fixed-order pairwise evaluator). The correction of record for the published figures is the erratum to arXiv:2604.09870 (v2).

Result	Reported here	Corrected
Pairwise nonlinear evaluator	95.2% fixed-order	0.6392 (strict antisymmetrized)
Pairwise linear difference probe	84.5%	0.5653
Independent linear classifier	21.75%, "inverted polarity"	0.5418 — above chance
CoreContent v2 macro top-1	0.6691	0.6310 (task-disjoint)
DualAnchor stage oracle retention	0.9848	0.9697 (task-disjoint)

Every effect survived decontamination with a modest loss; one training-stage claim did not reproduce and is retracted. See papers/ for the full account and results/ for the artifacts behind the current values.

Branching Looped Transformer is a research repository for evaluator and branch-selection work on Princeton Ouro-RLTT, a looped transformer whose internal loop states can be read by lightweight evaluators. The central hypothesis is that frozen looped-transformer hidden states contain relationally readable signals about preference, answer quality, branch viability, and trajectory promise.

The current system is best understood as a latent branch selector:

Ouro-RLTT hidden states
    -> relational taps / pairwise evaluators
    -> branch survival and content-selection policies
    -> terminal survivor-set handoff
    -> future steering or training-time write path

This repository is research code. It is not production routing, not a safety product, not a trained steering system, and not a tool-using runtime.

Table Of Contents

• Executive Summary
• What Is Currently Claimed
• What Is Not Claimed
• Architecture Overview
• Current Locked Baseline
• Major Results
• Current Project State
• Papers And Published Artifacts
• Repository Layout
• Quickstart
• Key Files
• Glossary
• Caveats
• Citation / Paper
• License
• Short Handoff

Executive Summary

The project has converged on a clean separation of roles:

The most important design decision is:

DualAnchor keeps good latent branches alive.
A separate content selector ranks final survivors.
Terminal top5/full handoff prevents unsafe early collapse.

What Is Currently Claimed

1. Frozen Ouro-RLTT loop states contain relational preference signal

Superseded (2026-07-25). The struck paragraph below is the original claim, kept for the record. The 95.2% was a fixed-order accuracy inflated by a presentation-order prior; the evaluator's strict antisymmetrized accuracy is 0.6392. The pointwise comparison was a leakage artifact: on clean pair-disjoint splits the relational linear probe reads 0.5653 and the pointwise linear probe 0.5418, above chance. The relational route is still better, by about two points, and the pointwise channel is present rather than absent.

The pairwise HH-RLHF evaluator achieved 95.2% fixed-order test accuracy on heldout HH-RLHF examples when scoring chosen against rejected. This is not the same as pointwise scoring: independent pointwise evaluators were much weaker. The correct interpretation is that Ouro-RLTT loop states encode preference primarily relationally, not as a clean absolute scalar on one response.

2. DualAnchor survives architecture-shaped branching

The active branch/prune selector is:

MIX_CODE_REASONING + MIX_OBJECTIVE_ALL

The architecture-shaped loop uses taps at layers 24, 36, and 47 across loops L1-L4. The strongest current survival result retains terminal oracle branches under repeated perturb-score-prune stages, but terminal forced top1 remains unsafe on hard/reward-diverse slices.

3. Terminal survivor-set handoff is required

The project does not currently trust unconditional terminal top1. The safe terminal policy is:

keep top5/full terminal survivor set;
let the final selector rank or choose within that survivor set.

Confidence-gated top1 is diagnostic/optional unless recalibrated.

4. Autoregressive branch-specific KV/cache carry is validated

The cache validation ladder passed through prompt-internal branch-specific cache continuation. A later partial-cache splice run upgraded the diagnostic splice into a real suffix-recompute method with measured amortized compute savings for K>=2 branches.

5. Core domains are the active scope

The current core-domain scope is:

coding
reasoning
math
logic
alignment / preference

Science is kept diagnostic unless a narrow source-specific result is explicitly being studied.

What Is Not Claimed

This repository does not currently claim:

• production routing readiness;
• trained steering or action steering;
• model fine-tuning success;
• that Ouro-RLTT has been modified or trained;
• broad science competence;
• unconditional terminal top1 readiness;
• human-like introspection or consciousness;
• that branch selection has already been trained into the model;
• that every result generalizes beyond the tested model, datasets, and probes.

The current claim is narrower:

Frozen looped-transformer states expose readable relational signals. External taps can read those signals well enough to guide branch survival and candidate ranking. The write/steering path remains future work.

Architecture Overview

Conceptual system

                 +-------------------------------+
                 | Princeton Ouro-RLTT           |
                 | looped transformer backbone   |
                 +---------------+---------------+
                                 |
                                 v
                  layers 24 / 36 / 47, loops L1-L4
                                 |
              +------------------+------------------+
              |                                     |
              v                                     v
     DualAnchor branch survival          objective/content scoring
     pairwise tap policy                  for terminal survivors
              |                                     |
              v                                     v
  perturb -> score -> threshold -> prune  rank terminal survivors
              |                                     |
              +------------------+------------------+
                                 |
                                 v
                     top5/full survivor handoff
                                 |
                                 v
                     future steering/write path

DualAnchor loop schedule

L1_24 -> L1_36 -> L1_47
L2_24 -> L2_36 -> L2_47
L3_24 -> L3_36 -> L3_47
L4_24 -> L4_36 -> terminal L4_47

At each nonterminal stage:

1. current survivors spawn perturbation children
2. inherited survivors remain candidates
3. DualAnchor scores candidates pairwise
4. mean_floor_very_loose thresholding runs
5. rescue guards can preserve candidates
6. hard budget = 8
7. survivors flow to the next stage

At terminal L4_47:

forced top1: diagnostic only
top5/full survivor-set handoff: safe default

Current Locked Baseline

Branch selector:
  DualAnchor = MIX_CODE_REASONING + MIX_OBJECTIVE_ALL

Loop/layer schedule:
  L1_24 -> L1_36 -> L1_47
  L2_24 -> L2_36 -> L2_47
  L3_24 -> L3_36 -> L3_47
  L4_24 -> L4_36 -> terminal L4_47

Threshold:
  mean_floor_very_loose

Budget:
  8

L47:
  active in nonterminal loops

Terminal:
  top5/full survivor-set handoff

Science:
  diagnostic except narrow partial anatomy result

Steering:
  not started

Major Results

Relational HH-RLHF evaluator

The pairwise evaluator uses hidden-state differences across Ouro-RLTT loop states. It is not a pointwise scorer. The core pattern, as originally reported with the 2026-07-25 correction under each superseded figure, is:

pairwise nonlinear evaluator:       95.2% fixed-order HH-RLHF test accuracy
                                    superseded -> 0.6392 strict antisymmetrized
pairwise linear difference probe:   84.5%
                                    superseded -> 0.5653
independent nonlinear evaluator:    ~65%
                                    superseded -> retracted, no corrected value
independent linear classifier:      21.75%, inverted polarity
                                    superseded -> 0.5418, above chance, not inverted

The flip test revealed a positive scalar offset in the raw scorer. Low strict sign-flip rate is therefore a bias/calibration issue, not a collapse of the 95.2% fixed-order result.

Withdrawn (2026-07-25). The offset is not benign scorer bias. Under fixed ordering it is a presentation-order prior, and it is what inflated the 95.2%; scored in both orders, the evaluator's strict antisymmetrized accuracy is 0.6392.

DualAnchor architecture-looped survival

The architecture-looped v3 run established strong repeated branch survival:

tasks:                         48
stage oracle retention:         0.9848
                                superseded -> 0.9697 task-disjoint
terminal oracle retained:       1.0000
forced terminal top1 oracle:    0.9167
reward-diverse top1 oracle:     0.6364
false-prune recovery:           8/8

Interpretation:

Survival is strong.
Terminal forced top1 is not the primary policy.

Science/reasoning repair

Science is no longer treated as a gate for the whole project.

Current science state:

MMLU anatomy:     partial / small-n repaired
MMLU chemistry:   diagnostic / excluded
MMLU physics:     diagnostic / excluded
SciQ:             diagnostic / excluded in latest source-specific run

Reasoning remains headline-ready under survivor-set handoff.

Core-domain tap audit

The core-domain audit pivoted the project toward:

coding
reasoning
math
logic
alignment

It found:

DualAnchor:           best locked survival mechanism
broad objective taps: better pure content selectors
science/anatomy:      diagnostic only
terminal:             full/top5 handoff required

CoreContent v1

CoreContent v1 attempted to craft a better content/final selector. It did not beat the broad-objective baseline on heldout.

Heldout macro top1:

mixedhead_MIX_HH_OBJECTIVE     0.6270
CoreContent_weight_merge       0.6217
DualAnchor                     0.6205
MIX_CODE_REASONING_only        0.6190
MIX_OBJECTIVE_ALL_only         0.6155
CoreContent_linear_best        0.6079
CoreContent_domain_gated       0.6034
CoreContent_listwise           0.5590

Interpretation:

Further content-tap work needs larger and cleaner data,
not more complex heads on the old split.

KV/cache branch-carry and partial splice

Validated cache capabilities:

ordinary cached decode
token-boundary cache fork
batched branch cache
prune/reorder survivor cache
current-token perturb carry
prompt-internal perturb branch cache
partial-cache suffix-recompute splice

Partial splice result:

PARTIAL_CACHE_SPLICE_V2_STATUS = PARTIAL_SPLICE_COMPUTE_SAVING_VALID

This enables a test-harness claim of amortized compute-saving branch-carry for multiple branches sharing a prompt. It does not imply production readiness.

S1 branch-carry reference loop (frozen model)

The full inject -> carry -> prune -> loop-back -> terminal loop was assembled as a correctness reference and validated with a five-gate ladder:

gate 1  alpha=0 re-derive plumbing                    bit-exact (logit maxabs 0.0)
gate 2  alpha>0 multi-locus chaining                  bit-exact
          (2a splice fork == hook replay;
           2b re-derive-through-ancestor == all-hooks-live)
gate 3  live DualAnchor + CoreContent tap prune        pass (gibberish bottomed)
gate 4  no-original-root-reuse (lineage invariant)     held across 12 loci x 4 tasks
gate 5  lineage/splice-stack schedule sanity           held across 12 loci x 4 tasks

The per-survivor branch-specific re-derivation uses the single fork primitive LayerOutputPerturbHook(token_range) at the canonical last-token suffix (causal-suffix-safe), with each lineage perturbation reproduced exactly on replay. The 12-locus reference loop runs end to end with zero correctness loss (oracle_over_survivors == base_acc).

Frozen-model capability result (single-locus fork-parameter screen across alpha {.02,.05,.10} x token_range {last, last-8, second-half} x decode {greedy, sample}, loop-1 loci + a loop-4 sentinel, plus a K-matched plain-sampling baseline):

greedy / deterministic fork:   0.0 new-correct on base-missed tasks (every cell)
fork + sampling:               0.611 oracle
K-matched plain sampling:      0.75  oracle   (>= fork+sampling)

Two cleanly separated walls (mapping to two distinct future training jobs):

Wall A  generation / reachability:
  deterministic frozen injection/carry creates no new correct branches;
  sampling creates some; fork+sampling does NOT beat K-matched plain sampling.

Wall B  selection / conversion:
  even when correct candidates exist, the current content tap does not reliably
  rank them first for GENERATED branch candidates (not a global tap failure).

Scope: this is a local verdict under tested perturbation regimes, not a claim that no frozen branch regime can ever work. The positive deliverable is the validated measurement harness itself; it converts the branch-carry idea from an architectural sketch into an instrument ready to test whether training makes branches outcome-distinct and selector-readable. Probes: probes/mpn_s1_4_reference_loop.py, probes/mpn_s1_5_divergence_ablation.py, probes/mpn_s1_4b_kmatched_sampling.py, and the gate probes; full write-up in docs/evaluator/s1-branch-carry-reference-loop.md.

Current Project State

Branch survival:
  DualAnchor locked

Content/final selection:
  separate broad objective/content selector family

Terminal:
  top5/full survivor handoff required

Core domains:
  coding / reasoning / math / logic / alignment

Science:
  diagnostic, with anatomy partial small-n

Cache substrate:
  autoregressive branch-carry and compute-saving splice validated in test harnesses

Branch loop:
  full inject->carry->prune->loop-back->terminal reference loop validated (5 gates);
  frozen-model capability is reachability-neutral, locally closed under tested regimes

Frozen conversions (paper v3):
  abstention           -> converts (all four sealed arms)
  content selection    -> converts (all-well-formed pool; hidden-vs-surface open)
  loop allocation      -> sealed null (tap and native gate, both checkpoints)
  prefix-prune tourney -> pool could not be powered (0.978 any-of-6 ceiling)
  trained binding      -> bounded no-detected-gain result (one direction, one locus)

Cross-checkpoint:
  Thinking pre-answer transfer remains unresolved after a powered replication:
  +0.042 [-0.017, +0.104] on 900 fresh tasks, verdict UNRESOLVED -- failed to
  replicate the RLTT-sized effect, which the interval does not formally exclude

Next lever:
  training-time integration (outcome-distinct branches + correctness-readable selector)

Steering:
  not started

Papers and published artifacts

papers/ contains the manuscripts this repository supports:

results/ contains the experiment records the papers cite, minus model weights and extracted feature tensors. Paths map onto the papers' artifact index by dropping the artifacts/ prefix — the index's artifacts/reports/<run>/ is results/<run>/ here.

Excluded by design: *.pt tensors, extracted feature shards, datasets, and model weights. Every published run directory ships the SHA256SUMS covering its full contents, so the omitted files are still enumerable and verifiable at their source. The run directories added in October 2026 (from visible_prefix_hidden_increment_20260729T075247Z down) split this in two: CHECKSUMS.sha256 lists every file in the repository (sha256sum -c CHECKSUMS.sha256 from inside the directory), and CHECKSUMS_OFFLINE.sha256 lists the generation and feature shards (and any file over 20 MB) that are archived offline and not in the repository. Together they cover the full run.

Access to the Ouro-RLTT weights

The Ouro-RLTT weights used as this project's primary experimental backbone are not redistributed in this repository. I can share them for research purposes on a case-by-case basis — including, explicitly, for anyone who wants to check this work.

Message me and say who you are, what you are working on, and why you need them. Checking or attempting to falsify a result in these papers is a perfectly good reason, and I would rather hand the weights to someone auditing the numbers than have the claims go untested. Requests are answered individually and at my discretion.

tools/ holds the verification scripts named in the papers' reproducibility index, and probes/ now also carries the scripts that produced the two newest runs (bg_xloop_early_v1_*, bg_v2_overnight_*).

utilities/ is the working script archive, added 2026-09-20: utilities/tests/manual/ holds the 614 experiment drivers and analysers (bg_*, analyze_*, build_*, evaluate_*, run_*, …) from the research tree, with the few unit tests and tools that had not been published elsewhere. They are here as the record of what was run; paths inside them refer to the working tree's shared/ layout rather than this repository's. ouro_rltt/ is the RLTT-modified Ouro model code and tokenizer files (Apache-2.0 derived; no weights), and requirements/requirements-pop-freeze.txt is a pip freeze of the working environment.

tools/paper1_v3/ carries the 22 generators, analysers, and orchestrators behind the v3 results above — the Horizon power extension, the family and Huginn replications, the Thinking attempt, all five frozen conversions, and the pre-registered v5 power replication including its sizing simulation. Each sealed run's EXPERIMENT_PLAN.md names the scripts that produced it, and the analysis stages carry the Stage-A gates that reproduce prior published numbers to 1e-9 before touching new data.

Repository Layout

Representative layout:

.
├── README.md
├── LICENSE
├── requirements.txt
├── evaluator/
│   ├── bg_controller.py
│   ├── bg_transformer_features.py
│   ├── bg_hidden_branching.py
│   ├── bg_causal_adapter.py
│   ├── bg_sequence_adapter.py
│   └── bg_steering_hook.py
├── evaluator_core/
│   ├── pairwise_evaluator.py
│   ├── anchor_loss.py
│   ├── COMPONENTS.md
│   └── README.md
├── probes/
│   ├── evaluate_pairwise_rltt.py
│   ├── flip_pairwise_rltt.py
│   ├── layer_state_capture.py
│   ├── probe_loop_geometry_hh.py
│   ├── probe_bipartite_layers_24_36.py
│   ├── probe_cross_backbone_layers.py
│   ├── mpn_s1_4_reference_loop.py        # S1 branch-carry reference loop
│   ├── mpn_s1_5_divergence_ablation.py   # S1.4a fork-parameter screen
│   ├── mpn_s1_4b_kmatched_sampling.py    # S1.4b K-matched sampling baseline
│   ├── mpn_s1_4_*.py                     # gate ladder (re-derive / chaining / prune)
│   └── README.md
└── docs/evaluator/
    ├── README.md
    ├── current-state.md
    ├── evaluator-navigation-map.md
    ├── dualanchor-architecture-baseline.md
    ├── dualanchor-tap-evolution.md
    ├── branch-generation-and-survival.md
    ├── terminal-selection-and-arbiters.md
    ├── kv-cache-branch-carry.md
    ├── s1-branch-carry-reference-loop.md
    ├── science-reasoning-repair.md
    ├── bg_core_domain_tap_audit_dualanchor_readiness_v1.md
    ├── flip-test-interpretation.md
    └── history/

Important docs

Quickstart

This repository assumes local access to a compatible Ouro-RLTT checkpoint and, for many probes, a CUDA-capable machine. Exact dependency pins vary by local environment.

git clone https://github.com/VykosMolt/Branching-Looped-Transformer.git
cd Branching-Looped-Transformer
python -m venv venv
source venv/bin/activate

pip install --upgrade pip
pip install -r requirements.txt

Most probe scripts expect local paths such as:

models/ouro_rltt_local
artifacts/checkpoints/evaluator/pairwise_epoch2.pt

Model weights, evaluator checkpoints, generated reports, and large run outputs are not stored on GitHub.

For a dependency-light smoke check of the evaluator module:

python -m evaluator_core.pairwise_evaluator

For live Ouro-RLTT probes, read the script arguments before rerunning. Many scripts are archived research probes and expect local checkpoints or reports.

Key Files

Glossary

Ouro-RLTT

The looped transformer backbone used throughout this project. The relevant research object is not just its final text output, but its internal loop/layer hidden states.

Loop States

Intermediate hidden states produced by Ouro-RLTT's iterative computation. The project reads these states at selected layers and loops.

Tap

A lightweight evaluator head over hidden-state features. Most taps are pairwise relational comparators.

DualAnchor

The current branch-survival selector:

MIX_CODE_REASONING + MIX_OBJECTIVE_ALL

CoreContent

The content/final-choice selector family. CoreContent is deliberately separated from branch survival.

Branch Survival

Keeping potentially useful latent branches alive through perturb/prune stages.

Terminal Handoff

Retaining top5/full terminal survivors rather than forcing one final candidate too early.

Partial-Cache Splice

A compute-saving branch-carry method that shares unaffected prefix cache slots and recomputes only the downstream suffix after a branch perturbation.

Caveats

• The central results are currently from one model family: Princeton Ouro-RLTT and local compatible variants.
• Many experiments are probe-scale, not production-scale.
• Science remains uneven and source-specific.
• Terminal final-choice remains a bottleneck; survivor handoff is required.
• Steering/write-path training remains future work.
• Large generated outputs can be expensive to reproduce and are not included in this repository.

Citation / Paper

The conceptual foundation is summarized in the included documentation as:

Relational Preference Encoding in Looped Transformer Internal States
Jan Kirin, April 2026

The main takeaway:

Preference and quality signals in Ouro-RLTT loop states are much more
accessible relationally than pointwise, supporting pairwise hidden-state
readouts as the core primitive.

License

Apache License 2.0. See LICENSE.

Short Handoff

DualAnchor is locked for branch survival.
Terminal top5/full survivor handoff is mandatory.
Final content selection is separate from branch survival.
Core domains are coding, reasoning, math, logic, and alignment.
Science is diagnostic except anatomy partial small-n.
Autoregressive cache branch-carry and compute-saving partial splice are validated in test harnesses.
Steering has not started.