Abstract
<title>Abstract</title> <p>Deriving a compute-optimal training recipe requires two choices that are usually made silently: what counts as a “parameter” when the loss surface is fitted, and what counts as “compute” when that surface is minimised under a budget. The two most cited compute-optimal studies made different choices — one counted non-embedding parameters, the other counted all of them — and both used the approximation C ≈ 6ND. This paper measures how much that bookkeeping matters. We train 35 decoder-only transformers on FineWeb-Edu, from 0.056M to 5.98M non-embedding parameters, at two BPE vocabulary sizes (V = 2048 complete, V = 8192 partial) and across a 5.8-fold range of training compute, with the learning rate calibrated by its own sweep, the cosine schedule matched to each run’s length, and no repeated data. We then fit the standard additive loss surface and derive the compute-optimal frontier four times over, changing only the accounting: parameters with or without embeddings, and compute as 6ND, as exact analytic FLOPs, or as measured device time. The output softmax — not attention — is what breaks 6ND at these scales, inflating the true cost by up to 8.5×. Holding the runs fixed, the fitted exponent a in N⋆ ∝ C a moves across 0.25–0.42 and the implied tokens-per-parameter ratio moves by a factor of 19, in the direction and close to the magnitude that a closed-form analysis predicts. Every bootstrap resample preserves the ordering, so the direction of the effect is certain even though each exponent is individually imprecise. A closed-form relation predicts each convention’s exponent from two elasticities computable from the architecture alone, and matches the fitted exponents to within 0.105. In a held-out test the conventions rank in the predicted order, the 6NneD convention being furthest from the measured optimum (20% versus 6%). The practical cost of the ambiguity is, however, small: because the compute-optimal valley is flat, following the wrong convention’s recommendation wastes at most 18% of the compute budget. Reported exponents are therefore not comparable across studies unless the accounting is stated, but the loss at stake is modest — and both facts are predictable in advance from two ratios that can be computed before any model is trained.</p>