Learning Discrete Cell and Niche Codes from Spatial Transcriptomics Using Dual Residual Vector Quantization

<jats:title>Abstract</jats:title> <jats:p> Spatially-resolved transcriptomics (SRT) measures gene expression at single-cell resolution while preserving each cell’s spatial location, enabling the joint study of <jats:italic>cell identity</jats:italic> and cellular <jats:italic>niche</jats:italic> , the recurring microenvironment that organises tissue function. Existing representation-learning methods typically capture only one of these axes at a time. We present SQUINT, a graph vector-quantized variational autoencoder (VQ-VAE) that learns <jats:italic>two</jats:italic> disjoint codebooks per cell from a shared architecture: a <jats:italic>cell codebook</jats:italic> quantising the per-cell embedding before neighbourhood aggregation, biased toward cell-intrinsic identity, and a <jats:italic>niche codebook</jats:italic> quantising the embedding after graph neural network (GNN) aggregation, biased toward spatial context. Both use residual vector quantization, giving a coarse-to-fine discrete-token hierarchy. SQUINT is trained with per-branch negative-binomial reconstruction objectives and three domain-motivated components that we show are crucial: a within-section cosine adjacency loss that anchors the niche codes in the spatial graph, a cross-section contrastive loss on the cell latents that aligns transcriptomically matched cells, and a decoder section covariate that absorbs batch effects. Across three datasets spanning four spatial assays (STARmap, MERFISH, CosMx, Xenium) and four tasks – niche identification, cell-type identification, cross-section integration, and spatial gene-expression imputation in held-out regions – SQUINT outperforms or is competitive with strong baselines on identification and achieves the most faithful cross-section integration. The resulting discrete vocabulary makes tissues directly consumable by transformer-style foundation models and enables one-step query-to-reference atlas mapping via code-distribution similarity, which we demonstrate on a CosMx human non-small-cell lung cancer cohort. </jats:p>