U-Space：揭示语言模型中的不确定性何时出现、为何出现

Explore

Explore the answers yourself

We asked the model three questions. Pick one, then click any word of its answer: the scene behind this page jumps to that word, and the numbers below show what the model's internal state said at that moment. Nothing here is hand-placed; it is all read from the model.

word·

leans toward·

cone alignment·

next-token entropy·

U-Lens at this word·

The four corners

Four sources of uncertainty, built from words

How do we know which corner is which? From words. We collected the words a model uses when it is unsure, sorted them into four groups, and contrasted each group with the words the model uses when it is certain. No examples had to be labelled by hand.

Solid chips are a few of the doubt anchors; dashed chips are a few of the certainty poles they are contrasted against. The full lists are in the repository under data/uspace_terms.yaml.

How it works

From a model's inner workings to a trust signal

1

Reading the inside of a model

While a language model writes, each layer of the network holds a long list of numbers: its internal state. On its own that list is unreadable. A recent tool called the J-Lens translates it: for any layer, it tells you which words that state is pushing the model toward. That makes the inside of a model something a person can read, and it is the foundation we build on.

2

Turning words of doubt into directions

Using the same translation in reverse, we take the words of doubt from the four groups and find the directions inside the model that correspond to them, each measured against the words of certainty. Four directions, one per kind of doubt: that is U-Space. Nothing is trained and no correct answers are needed to build it.

3

Following every word

For each word the model writes, we measure how much its internal state leans along each of the four directions. The four amounts place the word inside the pyramid. How strongly it leans overall tells us how much doubt there is at that word. This is the U-Lens: the path you flew through at the top of the page.

4

One number per answer, with a reason

When the model has finished reasoning, we combine how strongly its state leaned toward doubt with how spread out its word choices were. That gives one score for the whole answer: higher means more likely to be wrong. The corner the path favoured says what kind of doubt it was. All of it comes from the single pass that produced the answer.

For the technically minded: the formulas

lens_l(h) = unembed( J_l · h ), J_l = E[ ∂h_final / ∂h_l ] (the Jacobian lens)

A_cone = ‖ softplus( √d · ẑ ) ‖, ẑ = U^T ν(h) / ‖U^T ν(h)‖ (cone alignment at one token)

U-Lens = A_cone(end of thinking) × mean next-token entropy over the trace

U is the orthogonalised basis of the four doubt-minus-certainty contrasts pulled back through the lens; ν normalises the residual state. Full details and ablations are in the paper.

Does it work

Better than ten samples, at the cost of one

We gave three open models thousands of questions from four hard benchmarks and checked each answer. Then we asked every method to spot the wrong ones. The score is the usual one for this: 0.5 means guessing, 1.0 means perfect. "Length-matched" compares only answers of the same length, which removes a trick we explain below.

In practice

Decide what to trust and what to hand to an expert

Here are all the answers from one benchmark, lined up from least to most doubtful according to U-Lens. Teal answers turned out right, pink ones wrong. Drag the slider to choose how many of the most doubtful answers you would send to a human expert, and see what that buys you. These are the real scores from the paper.

·accuracy, accept all

·accuracy of accepted

·of the wrong answers deferred

·answers in this cell

And U-Space tells the reviewer where to start: for every answer handed over, which corner it leaned toward, so they know whether to look for a missing fact, a second reading of the question, or a conflict between sources.