Toward a General Theory of Representation Hypotheses
1University of Cambridge2University of Sydney3University of New South Wales
Choose a study to see its criteria, scale and protocol, and point at or tap a box for its definition.
Studies Apply Criteria at Different Points of One Lattice
The Platonic Representation Hypothesis (PRH) holds that increasingly capable models converge toward a common representation. However, recent studies have shown different views on aligning representations via their neighborhoods, spanning trees, and distances.
In the animation, the squared distances of model B are those of model A passed through the increasing link \(g_t\) of Theorem 3, which keeps the order of all distances.
Every neighbor set and the spanning tree therefore stay unchanged while CKA falls below one, so this pair of models can satisfy a neighbor criterion and fail a CKA criterion.
Model A
The link \(g_t\)
Neighbor Sets and Spanning Trees Lie on Two Branches below the Order of All Distances
An alignment criterion is a test that accepts a pair of models if one relation among shared evaluation samples, such as the neighbor sets in mKNN or the distances in CKA, agrees across the two models within a tolerance.
We call one criterion stronger than another if every pair accepted by the first is accepted by the second, and the criteria then form a lattice.
The numbers \(k\) of neighbors and \(m\) of samples set the scale of a criterion, i.e., the range of distances over which the two models are compared. The protocol of a criterion fixes the preprocessing, the combination of the scores of all layer pairs into one score, such as their maximum, and the calibration.
One pair of models can satisfy a criterion on one branch and fail a criterion on the other.
Continue with the four studiesConflicting Conclusions Can Come from the Scale, the Relation Compared, or the Protocol of a Criterion
PRHPlatonic Representation Hypothesis (Huh et al., 2024)
The convergence claimed by PRH amounts to agreement at the top of the lattice, whereas its evidence comes from mKNN, a criterion near the bottom that compares neighbor sets.
ARHAristotelian Representation Hypothesis (Gröger et al., 2026)
ARH restricts the claimed convergence to local neighborhoods because calibration keeps the size trend of mKNN but largely removes that of CKA. We show that this removal can come from calibrating a maximum over layers, whose chance level grows with effective dimension, so global convergence remains open.
Choose the layers compared, and press Replay to move each trend from raw to calibrated.
| Family | Layers | CKA, raw | CKA, calibrated | mKNN, raw | mKNN, calibrated |
|---|---|---|---|---|---|
| BLOOMZ | maximum | 0.89 | −0.02 | 1.00 | 1.00 |
| BLOOMZ | mid depth | 0.88 | 0.70 | 0.85 | 0.84 |
| LLaMA-1 | maximum | 0.68 | −0.85 | 0.41 | 0.38 |
| LLaMA-1 | mid depth | 1.00 | 0.97 | 0.82 | 0.82 |
| OpenLLaMA | maximum | 0.88 | 0.18 | 1.00 | 1.00 |
| OpenLLaMA | mid depth | 0.68 | 0.50 | 0.68 | 0.68 |
| Pythia | maximum | 0.93 | 0.89 | 1.00 | 1.00 |
| Pythia | mid depth | 0.98 | 0.96 | 0.99 | 0.99 |
BPCBack into Plato’s Cave (Koepke et al., 2026)
BPC infers coarse shared structure from the collapse of mKNN on larger evaluation sets. At a fixed \(k\), a larger evaluation set draws the boundary of the \(k\) nearest neighbors closer to each sample, so this collapse is at least partly a scale effect.
Enlarge the evaluation set, and choose whether \(k\) stays at 10 or grows as \(m/100\).
Model A near one anchor
Distances from the anchor
mKNN against \(m\)
SOGStructure over Geometry (You et al., 2026)
SOG infers converging global structure from the size trend of MST overlap. Every MST contains the edge from each sample to its nearest neighbor, and these edges make up about 88% of the shared tree edges on the studied vision–language pairs, so MST overlap mostly repeats one-nearest-neighbor overlap on these pairs.
Add noise to model B with the slider, or draw new samples.
The points of model B are those of model A plus independent noise of standard deviation \(d\) times the median nearest-neighbor distance in model A.
Tree of model A
Tree of model B
Overlap of the two trees
Forest edges in the tree of A…
…
The Training Procedure Decides the Criteria Met Reliably in a Linear Autoencoder, and the Objective in a Finite Contrastive Gibbs Model
Studies also claim that networks trained with different objectives learn similar representations, so we ask which criteria two independently trained models meet reliably, i.e., with high probability over their training runs.
Training procedureTheorem 2, linear autoencoder
Under gradient flow, the initialization of a linear autoencoder selects its gauge, i.e., an invertible change of the representation that its linear head undoes, so two runs disagree on distances at every sample size. Weight decay or tied weights remove this disagreement as the samples grow.
Choose a training procedure, and under gradient flow change the initialization of run B with the slider.
Gradient flow keeps the gauge set by the initial imbalance \(C=EE^\top-W^\top W\) between the encoder \(E\) and the decoder \(W\).
Run A
Run B
Agreement of the two runs
Projector onto the column space of the Gram matrixsame in both runs
ObjectiveTheorem 3, finite Gibbs model
In a finite contrastive Gibbs model whose population limit is the idealized contrastive learner of PRH, learners trained with InfoNCE and with the spectral contrastive loss learn one relation through different increasing functions. As their training samples grow, the two learners reliably agree on the order of all distances, whereas CKA between them tends to a limit below one.
We Test Our Explanation on Dozens of Pretrained Models and the Two Theorems on Thousands of Runs
The experimental record gives the full details and results of every experiment.
Pretrained vision and language models
At \(k = 10\), the median mKNN of the vision–language pairs falls from 0.091 to 0.0085 between 256 and 16,384 images, whereas at the same ratio \(k/m\) the medians for \(k = 10\) and \(k = 100\) agree within 4%.
Nearest-neighbor edges make up a median 0.875 and 0.889 of the shared MST edges at mid depth and at the last layer (panel (c) of the figure above). Beyond an affine fit, a fitted monotone link explains a further median of at most 0.0012 of the discrepancy between the squared distances of the two models.
In all four families, the calibration threshold of the layer maximum follows the largest random-pairing baseline, and on LLaMA-1 calibration turned the CKA trend from 0.68 into −0.85, a drop predicted from single-model features.
Under gradient descent without a regularizer, 0 of 48 pairs of runs agreed on neighbor sets, mean-normalized squared distances, or CKA at any finite sample size, whereas with weight decay, tied weights, or a small initialization, this agreement was reliable from 212 samples.
InfoNCE and spectral learners agreed exactly on the order of all distances among eight evaluation samples in 0 of 96 pairs up to 228 training examples and in 94 of 96 pairs at 234, near the size predicted before the runs.
Reference
A test that accepts a pair of models if one relation among shared evaluation samples agrees across the two models within a tolerance under a fixed scale and protocol.
Mutual \(k\)-nearest-neighbor overlap, i.e., the fraction of the \(k\) nearest neighbors of each sample shared by the two models, averaged over the samples.
The fraction of the edges of the minimum spanning tree (MST) shared by the two models.
Linear centered kernel alignment, i.e., the cosine similarity of the two centered Gram matrices, which depends on the distances of each model only up to rescaling.
A correction of a score by its values after random permutations of the samples of one model.
The expected value of a score after a random permutation of the samples of one model, i.e., its chance level.
The mean over vision models of the Spearman correlation between a score and the size of the language model.
Resources
BibTeX
@misc{shi2026plato,
title = {From Plato to Aristotle: Toward
a General Theory of Representation
Hypotheses},
author = {Shi, Dai and Han, Andi and Li,
Xiaoyu and Thompson, Luke and
Hern{\'a}ndez-Lobato, Jos{\'e} Miguel},
year = {2026},
note = {arXiv preprint, coming soon}
}
Studies discussed