Abstract
<title>Abstract</title> <p>Single-cell transformers are usually studied one checkpoint at a time, which makes it impossible to separate what the architecture imposes on a representation from what the training corpus writes into it. The three released scGPT checkpoints offer a controlled alternative: architecture, objective, optimizer settings and gene vocabulary are identical, and only the pre-training corpus differs (33M whole-human cells, 13.2M brain cells, 814K kidney cells). We use them as a common-origin divergence experiment. We first show that the three checkpoints were initialized from one shared random draw: 4,437 gene rows are bit-identical across all three, their values fall exactly in [−0.1, 0.1] with standard deviation 0.05770 (theory: 0.057735), and the singular values of that frozen block match the Marchenko–Pastur prediction to within 1%. This gives an exact noise model for everything else in the network. Using it, we find that roughly 80% of the initialization variance is still present in every gene-embedding matrix, and that only 119, 104 and 12 embedding directions rise above the initialization noise floor for the whole-human, brain and kidney checkpoints respectively. Once that floor is removed, organ adaptation turns out not to be a rotation or a rescaling of a shared set of biological axes. A similarity transform (translation, isotropic scale, rotation) explains 48% of the raw whole-human→brain drift; adding an independent scale per axis buys a further 0.3 percentage points; a full linear map reaches 57%. On the denoised learned geometry, a similarity transform explains only 29% (brain) and 8% (kidney), and only 20 of the 104 principal angles between the whole-human and brain learned subspaces fall below 45◦ . Conservation falls monotonically with spectral index, so the leading directions are the transferable ones. At gene level the direction of displacement, not its size, carries organ identity: the leading brain–kidney contrast axis tracks brain-versus-kidney expression enrichment (partial ρ = 0.246, p < 10−250), and this replicates on Tabula Sapiens. Across the 163 weight tensors, adaptation concentrates in the attention and feed-forward projections of the middle layers, while the 24 LayerNorm gain vectors move by only 4% on average and retain 99% of their initialization energy. Finally, only the brain checkpoint acquires a genuinely organ-specific axis: neural gene sets are over-encoded relative to the whole-human model (p = 7 × 10−37), whereas the kidney checkpoint shows no renal enrichment (p = 0.98). The practical consequence for programs that extract algorithms from foundation-model internals is direct: extract from the general checkpoint, and treat organ checkpoints trained on fewer than roughly ten million cells as mostly unlearned.</p>