ViTone: A Diagnostic Benchmark for Tonal Faithfulness in Vietnamese Text-to-Speech

<title>Abstract</title> <p>Vietnamese text-to-speech (TTS) is increasingly evaluated through global intel-ligibility and naturalness measures such as word error rate (WER), F0 error, mean opinion score (MOS), and speaker similarity MOS. These measures do not directly test whether a synthesised syllable preserves its intended lexical tone. This distinction matters because Vietnamese tone is contrastive and multi-cue: in the northern standard, six tones are distinguished through F0 contour and, for several tones, phonation cues such as laryngealisation. We present ViTone, a diagnostic benchmark and pre-registered evaluation protocol for auditing tonal faithfulness in Vietnamese TTS. ViTone combines tone-controlled minimal-pair stimuli generated from an explicit Vietnamese grapheme-to-phoneme rule set, an ASR-calibrated Tone Error Rate (TER) protocol, tone confusion matrices, and cue-level diagnosis separating F0 contour from laryngealisation. The protocol is designed to compare tonal faithfulness with naturalness under seen, unseen, northern-prompt, and southern-prompt conditions. This manuscript reports the benchmark design, the analysis plan, and a single-system pilot run that instantiates the full pipeline end to end on viXTTS with PhoWhisper-large scoring. The pilot fixes the ASR tone floor (0.40% overall on held-out human speech) and the cue-level classifier baseline (67.80% held-out accuracy, +1.04 pp from laryngealisation features), but it scores only 17 synthetic items from one system , so every cross-system and per-tone comparison is reported with its sample size and interval and is treated as descriptive. By making tonal contrast a first-class evaluation target, ViTone supports reproducible and linguistically grounded assessment of Vietnamese zero-shot and fine-tuned TTS systems.</p>