Duration / Speech Impl

[bkodes]

Reference: PR Comment

Ideas / Approach to incorporating duration / speech priors
--> e.g NaturalSpeech2 Impl

[bkodes]

@lucidrains just creating this discussion to jot ideas as we have them

[seastar105]

InstructTTS used variance adaptor from fastspeech2 as pitch/duration predictor with their own style encoder. this model is also nar model with neural audio codec. it generates discrete tokens with discrete diffusion.

will just using standard Text Encoder + pitch/duration predictor with prompt could work nicely? it seems much doable than training good text-to-semantic transformer.