EmoFI-TTS

Pretrained weights for EmoFI-TTS: Face-Conditioned Expressive Voice Synthesis with Intensity Modulation (arXiv:2610.30477).

A face image sets the emotion and its intensity, a reference utterance sets the speaker identity, and text sets the content. The model is a FastSpeech2 / Meta-StyleSpeech backbone with a single linearly projected style vector, followed by an intensity-conditioned refinement module and a HiFi-GAN vocoder. Code, training recipes, and evaluation scripts: github.com/luelhagos/Emofi.

Files

File	What it is	Place at (repo root)
refinement_15000.pth.tar	Released full system: backbone + refinement module	output/ckpt/MEAD_intensity/15000.pth.tar
backbone_30000.pth.tar	Backbone trained with cross-speaker pairing; starting point for refinement training	output/ckpt/MEAD/30000.pth.tar
warmstart.pth.tar	Pretrained backbone checkpoint that initializes jobs/train.sh	output/ckpt/pretrained/warmstart.pth.tar
generator_universal.pth.tar	HiFi-GAN universal vocoder	hifigan/
face_encoder.pt	Face encoder (ResNet-50, 768-d), needed only to embed new face images	face_encoder/
best_model.pt	Wav2Vec2 speech-emotion recognizer fine-tuned on MEAD, used for emotion and intensity evaluation	ser_judge/

Download everything with:

hf download luel/emofi-tts --local-dir .

Training and evaluation on MEAD also need the preprocessed data in luel/emofi-mead.

Results (speaker-disjoint MEAD test set, intensity level predicted from the face)

	WER (%)	UTMOS	L1	L2	L3	Intensity acc.
EmoFI-TTS	6.9	3.14	76.4	42.8	74.1	64.4
Real recordings	6.6	2.86	69.7	34.6	80.3	61.5

Objective results from Tables 1 and 2 of the paper. L1 to L3 are per-level intensity recall from the released speech-emotion recognizer; see the paper for the human evaluation and the baselines.

Intended use and limitations

Research use. Trained on MEAD only: six emotions (neutral, angry, happy, sad, surprised, fear) at three intensity levels, English, studio recordings, with a speaker-disjoint test split. Voice transfer to speakers and recording conditions outside MEAD is not evaluated and should not be assumed. Do not use to imitate a person's voice without consent.

License and credits

Code and the TTS checkpoints are released under the MIT License. The HiFi-GAN universal generator is from the HiFi-GAN authors (MIT).

Citation

@article{jeon2026emofi,
  title   = {EmoFI-TTS: Face-Conditioned Expressive Voice Synthesis with Intensity Modulation},
  author  = {Jeon, Yejin and Beyene, Luel H. and Maltais, Marie and Kim, Hyounghun and Adelani, David I.},
  journal = {arXiv preprint arXiv:2610.30477},
  year    = {2026},
  url     = {https://arxiv.org/abs/2610.30477}
}