Abstract
<title>Abstract</title> <p>Progressive width growth can save compute cost by expanding smaller model checkpoints. Each expansion step is a difficult transition that needs to balance two competing goals: preserving the behaviors previously learned by the model and improving the new block’s learning efficacy. We propose an offline growth recipe with three components. First, the Small Residual Integration Operator (SRIO) applies a closed-form orthogonal rotation at both the input and output residual boundaries. The operator at the block inputs rotates the new subspace’s all-ones component toward the old subspace, improving the old weight’s read-strength for the new inputs. The output operator is the inverse. Second, we use block-specific static expansion policies for the attention and the feed-forward network (FFN) blocks: convex expansion for attention and tiled-copy for FFN. Additionally, we add a scalar warmup gate that gradually introduces the newly added attention and FFN outputs. The block-specific expansion and the SRIO operator have the strongest positive impact amoung the three part recipe. In matched 256->384 width growth experiments, the complete recipe reaches a training loss of 4.2111, compared to 4.2527 for the no-operator baseline and 4.2606 for the LiGO controls reimplemented in the same codebase. Multi-stage runs further demonstrate repeated growth from 24M to 120M parameters. The SRIO and the block-specific expansion are folded into the grown model weights offline and therefore have no persistent runtime operator during training or inference; only the scalar warmup gate is active briefly after growth.</p>