Knowing Before Saying · interactive companion

Does the model know it is wrong before it says it is sure?

Every number here is from the paper. Nothing is simulated. Four open models, four tasks, one linear probe on the hidden state at seventy percent of depth, scored against an external grade with AUROC.

Sixteen cells

Each cell compares two readouts of the same attempts: the confidence the model wrote down, and what a probe reads off its hidden state. A terracotta edge marks the cell where the probe wins.

stated confidencelinear probeprobe winstie
12probe wins
1tie
3stated wins
+0.10mean gap
+0.20gap on MATH

Erase the signal and watch the confidence go blind

On Mistral, take the correctness direction the probe found and scale it at the confidence token, after the answer is written. At gain 1 the model is untouched. At gain 0 the direction is erased. Drag the slider.

correctness directionrandom direction, magnitude matched

gain
1×
stated-confidence AUROC0.84
random direction0.84
mean confidence, right answers96
mean confidence, wrong answers84
records150 held out

Mistral-Small-24B on MATH, direction fitted on a disjoint half, applied at layer 28. The random direction is scaled to the same magnitude. At gain 4 the random control stops parsing as a confidence, so it has no point there. Single seed; the paper's bootstrap interval on the erasure drop is 0.12 to 0.42.

Paper (PDF) · Poster · Back to the site