PsarAI-S2S

Speech and text in Khmer (ខ្មែរ), English and French with one model: speech recognition, translation, fast speech output, and speech-to-speech translation. Mixed speech (Khmer with English or French in the same sentence) needs no language setting.

Task Method Languages
asr speech → text MMS-1B (wav2vec2, CTC) with a code-switching adapter km, en, fr and any mix
translate text → text NLLB-600M fine-tuned on km ↔ en ↔ fr km, en, fr
speak text → speech NLLB encoder → non-autoregressive unit decoder → unit HiFi-GAN (24 kHz) km, en
s2st speech → text + speech to Khmer: speech straight into the translator (no transcript in between); to English: ASR → MT → km, → en

Everything is in one model.safetensors (3.7 GB, fp16). The voice is a single synthetic voice ("Jarvis").

Usage

import torch, soundfile as sf
from transformers import AutoModel

model = AutoModel.from_pretrained("nphearum/PsarAI-S2S", trust_remote_code=True, dtype=torch.float16).to("cuda")

text = model.asr("speech.wav")                          # Khmer / English / French / mixed
text = model.asr("speech.wav", language="fra")          # "eng" or "fra": best accuracy for one language
out = model.asr("speech.wav", return_timestamps=True)  # {"text", "words": [{"word", "start", "end"}]} in seconds
text = model.asr("speech.wav", lm=True)                 # more accurate: CTC beam search + 6-gram KenLM (below)
model.translate("Good evening, sir.", to="km")          # -> សាយ័ណ្ហសួស្តី លោក។   (to: km | en | fr)
wav = model.speak("សួស្តី លោក។", lang="km", pace=1.15)  # numpy float32 at model.sample_rate (24 kHz)
text, wav = model.s2st("speech.wav", to="km")
sf.write("out.wav", wav, model.sample_rate)

Word timestamps come from the CTC alignment (20 ms frames). Khmer is written without spaces between words and the model puts spaces between phrases, so Khmer timestamps are per phrase; English and French are per word.

Language model (optional)

By default asr uses greedy CTC decoding. With lm=True it runs a CTC prefix beam search scored by a character 6-gram KenLM (lm/chars6.arpa, 79 MB, downloaded on first use). It works per character, so it can fix spelling inside unspaced Khmer text, and it covers Khmer, English, French and mixed speech. It was trained on about 830k lines of km/en/fr text (modified Kneser-Ney); sentences that overlap the test sets were removed.

# pip install https://github.com/kpu/kenlm/archive/master.zip
text = model.asr("speech.wav", lm=True)                 # recommended: alpha=0.5, beta=0.5, beam_width=16 (defaults)
text = model.asr("speech.wav", lm=True, beam_width=32)  # slightly better, ~1.8x slower
text = model.asr("speech.wav", lm="my.arpa")            # your own character KenLM (ARPA or binary)

Recommended settings. The defaults (alpha=0.5 LM weight, beta=0.5 length bonus, beam_width=16) were tuned on FLEURS validation (km/en/fr). Content CER % on FLEURS test (100 clips each) and the mixed set:

Setting km en fr mixed Time
defaults (0.5 / 0.5 / 16) 9.26 4.64 4.33 1.57 1x
beam_width=32 9.19 4.63 4.35 1.54 ~1.8x
alpha=0.6, beam_width=32 9.09 4.79 4.46 1.58 ~1.8x
beam_width=8 9.62 4.79 4.40 1.57 ~0.5x

Keep the defaults for general or mixed speech, use beam_width=32 when speed does not matter, and alpha=0.6, beam_width=32 for Khmer-only audio.

Content CER %, language="auto", the same clips with both methods (100 clips per FLEURS set):

Test set greedy lm=True
Khmer–English/French mixed held-out (300) 2.18 1.57
FLEURS Khmer test 12.39 9.26
FLEURS English test 5.07 4.64
FLEURS French test 5.00 4.33

The beam search runs on the CPU and takes about 2x as long as greedy decoding (FLEURS: 0.1 s vs 0.05 s per clip on an RTX 3090). It does not return timestamps; use return_timestamps=True without lm for those.

Requirements: transformers, torch, soundfile, soxr. Audio input: a file path, or a numpy array with sampling_rate=. pace > 1 makes the voice slower (it speaks a little fast by default).

Results

Speech recognition, content CER % (letters only), greedy, compared with PsarAI-ASR-1B (V1):

Test set V1 PsarAI-S2S asr with language=
grkpp, Khmer news, new speakers 13.72 9.89
khmer10, unseen speaker 4.84 3.44
DDD held-out (clips not in training) 5.58 4.10
FLEURS Khmer test 14.05 13.04
FLEURS English test 4.74 4.64 3.89 (eng)
FLEURS French test — 4.71 3.51 (fra)
Jarvis Khmer–English mixed (synthetic) 1.83 2.18
Jarvis Khmer–French mixed (synthetic) 21.23 2.60

Translation, chrF on 200 held-out segments per direction (higher is better), before → after fine-tuning: km→en 73.2 → 80.3, km→fr 64.8 → 74.0, en→km 77.0 → 82.4, en→fr 82.5 → 87.4, fr→km 68.9 → 76.6, fr→en 83.0 → 86.6 (mean 74.9 → 81.2). The references are machine translations of Khmer web text, so this measures agreement with them, not human-judged quality.

Speech output (speak), intelligibility: speech is generated from held-out Jarvis transcripts and transcribed back by ASR (CER, lower is better; the original recordings score 0.017 / 0.004 / 0.030): Khmer 0.055, English 0.042, mixed 0.189. Speed on an RTX 3090: about 0.01 s of compute per second of speech (a 5 s sentence in about 0.05 s); about 1 s on CPU.

Speech → Khmer (s2st(to="km")) on held-out Jarvis clips: CER 0.010 (Khmer speech), 0.037 (English), 0.100 (mixed) against reference translations; 0.6–1.3 s per clip on GPU.

How it works

  • ASR: the MMS-1B base stays frozen; small adapters (~2.3M parameters each) and output heads are trained. The default adapter covers Khmer, English, French and code-switching (vocabulary: Khmer script, Latin letters, French accents, | word separator). eng and fra adapters are the MMS ones for single-language speech.
  • Speech → Khmer uses a 9M-parameter bridge that maps all 49 ASR layers into the translator's input space, so the translator reads speech directly instead of a transcript (this avoids passing ASR mistakes on). It was trained with the original khmer-translate-600m decoder, which is kept for it (bridge_decoder).
  • Speech output: a non-autoregressive decoder predicts discrete speech units (mHuBERT-147, k-means 1,000) and their durations from the translator's encoder states; a HiFi-GAN turns units into 24 kHz audio.

Limitations

  • One synthetic voice that imitates a fictional character's voice; the rights to that character's voice belong to their owners. No voice cloning in this model.
  • Speech output is Khmer and English only; French output is not trained yet. Long or rare words can come out blurred; mixed sentences are less clear than single-language ones.
  • Speech → English goes through ASR, so recognition errors reach the translation. Speech → Khmer does not.
  • Domain: speech output and the mixed-speech tests use synthetic "Jarvis" speech; real Khmer is tested on grkpp, DDD and FLEURS. The translation training data are machine translations of web text.
  • Khmer លោក ("you", polite) is sometimes translated as "he".
  • Output of the ASR is lowercase; spaces in Khmer follow phrase boundaries, not word boundaries.

License and credits

CC-BY-NC-4.0 (non-commercial), following facebook/mms-1b-all and NLLB-200. Speech units: utter-project/mHuBERT-147. Training data: Digital-Divide-Data/khmer-speech-dataset (CC-BY-SA-4.0), google/fleurs (CC-BY-4.0), Common Voice French via ele-sage/french-asr-quebec-eu (CC0), nphearum/dataset-asr-tts (CC-BY-NC-4.0), and machine-translated nphearum/khmer-raw-text-3M text. Check that your use is permitted before using this model.

Downloads last month
92
Safetensors
Model size
2B params
Tensor type
F16
·
Inference Providers NEW
This model isn't deployed by any Inference Provider. 🙋 Ask for provider support

Model tree for nphearum/PsarAI-S2S

Finetuned
(443)
this model
Quantizations
2 models