Instructions to use nphearum/PsarAI-S2S with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- Transformers
How to use nphearum/PsarAI-S2S with Transformers:
# Use a pipeline as a high-level helper from transformers import pipeline pipe = pipeline("automatic-speech-recognition", model="nphearum/PsarAI-S2S", trust_remote_code=True)# pip install -U transformers accelerate # Load model directly from transformers import AutoModel model = AutoModel.from_pretrained("nphearum/PsarAI-S2S", trust_remote_code=True, device_map="auto") - Notebooks
- Google Colab
- Kaggle
PsarAI-S2S
Speech and text in Khmer (ខ្មែរ), English and French with one model: speech recognition, translation, fast speech output, and speech-to-speech translation. Mixed speech (Khmer with English or French in the same sentence) needs no language setting.
| Task | Method | Languages |
|---|---|---|
asr speech → text |
MMS-1B (wav2vec2, CTC) with a code-switching adapter | km, en, fr and any mix |
translate text → text |
NLLB-600M fine-tuned on km ↔ en ↔ fr | km, en, fr |
speak text → speech |
NLLB encoder → non-autoregressive unit decoder → unit HiFi-GAN (24 kHz) | km, en |
s2st speech → text + speech |
to Khmer: speech straight into the translator (no transcript in between); to English: ASR → MT | → km, → en |
Everything is in one model.safetensors (3.7 GB, fp16). The voice is a single synthetic voice ("Jarvis").
Usage
import torch, soundfile as sf
from transformers import AutoModel
model = AutoModel.from_pretrained("nphearum/PsarAI-S2S", trust_remote_code=True, dtype=torch.float16).to("cuda")
text = model.asr("speech.wav") # Khmer / English / French / mixed
text = model.asr("speech.wav", language="fra") # "eng" or "fra": best accuracy for one language
out = model.asr("speech.wav", return_timestamps=True) # {"text", "words": [{"word", "start", "end"}]} in seconds
text = model.asr("speech.wav", lm=True) # more accurate: CTC beam search + 6-gram KenLM (below)
model.translate("Good evening, sir.", to="km") # -> សាយ័ណ្ហសួស្តី លោក។ (to: km | en | fr)
wav = model.speak("សួស្តី លោក។", lang="km", pace=1.15) # numpy float32 at model.sample_rate (24 kHz)
text, wav = model.s2st("speech.wav", to="km")
sf.write("out.wav", wav, model.sample_rate)
Word timestamps come from the CTC alignment (20 ms frames). Khmer is written without spaces between words and the model puts spaces between phrases, so Khmer timestamps are per phrase; English and French are per word.
Language model (optional)
By default asr uses greedy CTC decoding. With lm=True it runs a CTC prefix beam search scored by a
character 6-gram KenLM (lm/chars6.arpa, 79 MB, downloaded on first use). It works per character, so it can fix
spelling inside unspaced Khmer text, and it covers Khmer, English, French and mixed speech. It was trained on about
830k lines of km/en/fr text (modified Kneser-Ney); sentences that overlap the test sets were removed.
# pip install https://github.com/kpu/kenlm/archive/master.zip
text = model.asr("speech.wav", lm=True) # recommended: alpha=0.5, beta=0.5, beam_width=16 (defaults)
text = model.asr("speech.wav", lm=True, beam_width=32) # slightly better, ~1.8x slower
text = model.asr("speech.wav", lm="my.arpa") # your own character KenLM (ARPA or binary)
Recommended settings. The defaults (alpha=0.5 LM weight, beta=0.5 length bonus, beam_width=16) were tuned
on FLEURS validation (km/en/fr). Content CER % on FLEURS test (100 clips each) and the mixed set:
| Setting | km | en | fr | mixed | Time |
|---|---|---|---|---|---|
| defaults (0.5 / 0.5 / 16) | 9.26 | 4.64 | 4.33 | 1.57 | 1x |
beam_width=32 |
9.19 | 4.63 | 4.35 | 1.54 | ~1.8x |
alpha=0.6, beam_width=32 |
9.09 | 4.79 | 4.46 | 1.58 | ~1.8x |
beam_width=8 |
9.62 | 4.79 | 4.40 | 1.57 | ~0.5x |
Keep the defaults for general or mixed speech, use beam_width=32 when speed does not matter, and
alpha=0.6, beam_width=32 for Khmer-only audio.
Content CER %, language="auto", the same clips with both methods (100 clips per FLEURS set):
| Test set | greedy | lm=True |
|---|---|---|
| Khmer–English/French mixed held-out (300) | 2.18 | 1.57 |
| FLEURS Khmer test | 12.39 | 9.26 |
| FLEURS English test | 5.07 | 4.64 |
| FLEURS French test | 5.00 | 4.33 |
The beam search runs on the CPU and takes about 2x as long as greedy decoding (FLEURS: 0.1 s vs 0.05 s per clip on an RTX 3090).
It does not return timestamps; use return_timestamps=True without lm for those.
Requirements: transformers, torch, soundfile, soxr. Audio input: a file path, or a numpy array with
sampling_rate=. pace > 1 makes the voice slower (it speaks a little fast by default).
Results
Speech recognition, content CER % (letters only), greedy, compared with PsarAI-ASR-1B (V1):
| Test set | V1 | PsarAI-S2S asr |
with language= |
|---|---|---|---|
| grkpp, Khmer news, new speakers | 13.72 | 9.89 | |
| khmer10, unseen speaker | 4.84 | 3.44 | |
| DDD held-out (clips not in training) | 5.58 | 4.10 | |
| FLEURS Khmer test | 14.05 | 13.04 | |
| FLEURS English test | 4.74 | 4.64 | 3.89 (eng) |
| FLEURS French test | — | 4.71 | 3.51 (fra) |
| Jarvis Khmer–English mixed (synthetic) | 1.83 | 2.18 | |
| Jarvis Khmer–French mixed (synthetic) | 21.23 | 2.60 |
Translation, chrF on 200 held-out segments per direction (higher is better), before → after fine-tuning: km→en 73.2 → 80.3, km→fr 64.8 → 74.0, en→km 77.0 → 82.4, en→fr 82.5 → 87.4, fr→km 68.9 → 76.6, fr→en 83.0 → 86.6 (mean 74.9 → 81.2). The references are machine translations of Khmer web text, so this measures agreement with them, not human-judged quality.
Speech output (speak), intelligibility: speech is generated from held-out Jarvis transcripts and
transcribed back by ASR (CER, lower is better; the original recordings score 0.017 / 0.004 / 0.030):
Khmer 0.055, English 0.042, mixed 0.189. Speed on an RTX 3090: about 0.01 s of compute per second of
speech (a 5 s sentence in about 0.05 s); about 1 s on CPU.
Speech → Khmer (s2st(to="km")) on held-out Jarvis clips: CER 0.010 (Khmer speech), 0.037 (English),
0.100 (mixed) against reference translations; 0.6–1.3 s per clip on GPU.
How it works
- ASR: the MMS-1B base stays frozen; small adapters (~2.3M parameters each) and output heads are trained.
The default adapter covers Khmer, English, French and code-switching (vocabulary: Khmer script, Latin letters,
French accents,
|word separator).engandfraadapters are the MMS ones for single-language speech. - Speech → Khmer uses a 9M-parameter bridge that maps all 49 ASR layers into the translator's input space,
so the translator reads speech directly instead of a transcript (this avoids passing ASR mistakes on).
It was trained with the original khmer-translate-600m decoder, which is kept for it (
bridge_decoder). - Speech output: a non-autoregressive decoder predicts discrete speech units (mHuBERT-147, k-means 1,000) and their durations from the translator's encoder states; a HiFi-GAN turns units into 24 kHz audio.
Limitations
- One synthetic voice that imitates a fictional character's voice; the rights to that character's voice belong to their owners. No voice cloning in this model.
- Speech output is Khmer and English only; French output is not trained yet. Long or rare words can come out blurred; mixed sentences are less clear than single-language ones.
- Speech → English goes through ASR, so recognition errors reach the translation. Speech → Khmer does not.
- Domain: speech output and the mixed-speech tests use synthetic "Jarvis" speech; real Khmer is tested on grkpp, DDD and FLEURS. The translation training data are machine translations of web text.
- Khmer លោក ("you", polite) is sometimes translated as "he".
- Output of the ASR is lowercase; spaces in Khmer follow phrase boundaries, not word boundaries.
License and credits
CC-BY-NC-4.0 (non-commercial), following facebook/mms-1b-all and NLLB-200. Speech units: utter-project/mHuBERT-147. Training data: Digital-Divide-Data/khmer-speech-dataset (CC-BY-SA-4.0), google/fleurs (CC-BY-4.0), Common Voice French via ele-sage/french-asr-quebec-eu (CC0), nphearum/dataset-asr-tts (CC-BY-NC-4.0), and machine-translated nphearum/khmer-raw-text-3M text. Check that your use is permitted before using this model.
- Downloads last month
- 92