Request for Training Code and Reproduction Guidance

[UkiTenzai]

I attempted to train the AUV model using only speech signals (without music and general audio) with the official DescriptAudioCodec(DAC) trainer. The training setup includes reconstruction loss, feature loss, vq loss, and adversarial loss, but I excluded the distillation loss since my focus is solely on speech signals. After training for 100k iterations, the results were bad (PESQ score of only 1.4), far below the pretrained model’s performance (PESQ 2.4) provided in the repository. I also tried training with the trainer from AcademicCodec but encountered similar issues.

Could you please provide the training code for reference?

Additionally, I noticed that the paper mentions BigCodec and StableCodec. Would it be feasible to replicate the training process using their trainers?

[SWivid]

Hi @UkiTenzai , sorry for the late response, I forgot to turn on the github watcher of this repo.

Yes, I think the trainer from BigCodec, or more closely, the one from X-Codec2 will be better.
(p.s. to avoid any misunderstanding, we are not using exactly the same codebase, will see if possible to release).

The setup of discriminators are almost the same except that they use this for MS-STFT discriminator:

fft_sizes: [78, 126, 206, 334, 542,876,1418,2296]
hop_sizes: [39, 63, 103, 167, 271,438,709,1148]
win_lengths: [78, 126, 206, 334, 542,876,1418,2296]

but we use this:

"n_ffts": [206, 334, 542, 876, 1418, 2296],
"hop_lengths": [103, 167, 271, 438, 709, 1148],
"win_lengths": [206, 334, 542, 876, 1418, 2296],

[SWivid]

For speech-only case, refer to Table 3, trained with LibriSpeech 960h:

• PESQ-WB will be 2.18 at 400k, without distill ((A1) model), and 2.09 at 200k with no-ema ckpt (not mentioned in paper).
• PESQ-WB will be 2.14 at 200k, with distill ((A3) model).

Also, two points may help:

1. we use EMA, will have better performance. See the trainer of F5-TTS or also SAC.
2. Initialization is a dominant issue (if bad luck; though we have no deterministic way to suppress yet). The resulting scores reported in tables of our paper is the avg of three runs.