Abstract
<title>Abstract</title> <p>Sparse Mixture-of-Experts (MoE) models suffer from two persistent issues in incremental training: expert output homogenization (Sim ≈ 1.00) rendering sparse computation wasteful, and catastrophic language drift (rising PPL) under continued data exposure. Existing remedies rely on auxiliary orthogonal regularization or knowledge distillation from teacher models—a purely architectural solution without performance penalty remains absent. We propose a three-component synergistic system: FrozenPath (a frozen copy of the original FFN serving as a representation anchor), Data Shard (text-level shard binding, where complete documents are permanently assigned to dedicated experts), and Dual-Loop (a within-expert dual forward self-reinforcement cycle). Five ablation configurations are evaluated on Qwen2.5-0.5B (24 layers, MoE-4 experts, 15,000 steps, FineWeb, LR = 5 × 10−5): E (all three components), A (no Dual-Loop), G2 (three loops), I (no Data Shard, random routing), C (no FrozenPath). The full configuration E achieves PPL = 20, HellaSwag = 38.5%, TQA MC2 = 42.8%, substantially outperforming the dense baseline (PPL = 143). Ablation reveals independent contributions: FrozenPath is a survival necessity — removing it causes PPL to explode 85× to 1318 with garbled generation; Data Shard is the sole source of expert differentiation — Sim = 0.85 with sharding vs. Sim = 1.00 without (PPL indistinguishable, Sim perfectly separates the two); Dual-Loop provides a marginal gain of +2 PPL and +2% HellaSwag, with 2-to-3 loop saturation (G2 PPL = 21 vs. E = 20). On the deployment side, FP16 single-expert inference occupies 3.5 GB VRAM with zero PPL drift, and a FastAPI inference server achieves Level 4 engineering maturity at 7/10. The FrozenPath + Data Shard + Dual-Loop system provides a distillation-free, auxiliary-lossfree, low-drift, low-homogenization MoE training framework. Each component has a distinct responsibility: FrozenPath locks the PPL floor, Data Shard drives expert differentiation, and Dual-Loop fine-tunes convergence. The system is naturally suited for enterprise vertical incremental fine-tuning, multi-domain mixed MoE training, and long-cycle small-model iteration.</p>