How do LLMs think about thinking?

Mei Chen

throw Pebble a ball by tapping anywhere on the screen

Research

Open questions

  • How can we trust a model when it self-reports? Can this be verified?

Work

Probe AUROC by layer depth on MATH for four open models, with the fixed layer marked
Accepted · MATH-AI workshop, NeurIPS 2026 · with Gordon Stein and Andreas Ziegler

Knowing Before Saying

A linear probe on the hidden state predicts whether an answer is right better than the model's stated confidence does, in 12 of 16 cases across four open models. On one model, erasing that direction collapses the stated confidence too. The model knows more than it says, and sometimes uses it.

Paper (PDF)Explore the cellsPoster

Strict pass rate against cost per scenario, raw versus scaffolded, with the Pareto frontier
Talk at AIME-Con 2026 · poster at MATH-AI, NeurIPS 2026 · with Gordon Stein

GeoGenBench

Can a language model draw a geometry diagram? The model describes the construction, an engine draws it, and every prompt ships with its own key, so a compiler grades the result instead of a vision model. Graded this way, the scaffold matters more than the model.

DatasetTry the checkerExplainerPosterSlides

The same points on the top PCA axes, where classes interleave, and on the probe axis, where they separate
Lab book · complete, unpublished · June 2026

Where a concept lives

Consciousness prompts against matched philosophy controls in a small reasoning model. A late-layer direction separates them almost perfectly, in a direction so low in variance that PCA never shows it. A scatter plot of this data would have produced a false null.

Explore layer by layerAUC by layerNeighbours

In progress

  • Designed · no data yet

    Introspective grounding. When a model says it is uncomfortable, is anything inside it actually changing? I nudge a candidate direction up and down, keep everything else fixed, and watch whether the words follow.

  • Designed · no data yet

    Welfare signal dissociation. Do what a model does, what it says, and what shows up inside it line up? I am mapping where they agree and where they quietly split, across models.

  • Instrument · running

    Simulation observatory. Small social games between AI agents with hidden incentives. I keep each agent's private notes, predictions, and public words side by side, and watch what happens when they drift apart.

Play with the ideas

The first three use the real numbers and activations behind the papers. Nothing in them is simulated.

Earlier

Brain-computer interfaces, 2017 to 2019: tactile EEG across age groups, paper; mindfulness and BCI accuracy, paper; music imagery as a control signal, paper.