VimeML Tiny Chinese GPT v1
A 20,036,736-parameter Simplified Chinese decoder-only language model for local contextual reranking of pinyin input-method candidates and short phrase prediction. This release contains the vimeml-tiny-zh-v1 step 844,730 FP32 inference bundle, its INT8 block32, FP32-compute Core ML stateless package, and a matching chunked KV-cache Core ML package.
Status: release for iOS deployment. The Vime input method on iOS uses the model to rerank pinyin candidates. The Core ML packages pass the Mac alignment gates. They have been measured on iPhone 16 Pro Max in a test host and inside the Vime keyboard extension, where the model reranks pinyin candidates (see Performance).
This is a custom PyTorch model, not a Transformers AutoModel integration or an instruction/chat model. SentencePiece, candidate retrieval (librime with rime-ice), candidate scoring and search run outside the Core ML graph. The architecture, loader and scoring code are included under source/; no trust_remote_code or training checkpoint is required.
Files and identity
| Path | Contents |
|---|---|
inference/ |
Unique FP32 state_dict, config, tokenizer and provenance manifest; no optimizer |
coreml/ios18-int8-block32/ |
Stateless Core ML .mlpackage and manifest |
coreml/ios18-int8-block32-kvchunk/ |
Chunked KV-cache Core ML .mlpackage and manifest |
coreml/ios18-int8-block32-kvbatch/ |
Batched single-step KV Core ML .mlpackage and manifest (pinyin beam search) |
coreml/ios18-int8-block32-kvbatch-v2/ |
Batched KV v2 Core ML .mlpackage and manifest: multi-token prefill and decode in one package, scored head (pinyin beam search) |
coreml/ios18-int8-block32-kvsequence/ |
Batched KV sequence-head Core ML .mlpackage and manifest: log-probability of a given token at every new position, so a batch of known continuations is scored in one call (sentence scoring) |
coreml/ios18-int8-block32-kvbatch-v3/ |
Batched KV v3 Core ML multifunction .mlpackage and manifest: logits, scored and score functions over one weight copy, the prompt passed once as a shared prefix (pinyin beam search and sentence scoring; recommended) |
source/ |
Frozen VimeML Python source, scripts, configurations and requirements |
infer.py |
FP32 candidate-scoring and phrase-prediction wrapper using the zh-v1 bundle loader |
evaluation/summary.json |
Aggregate quality, alignment and measurement results; no benchmark texts or traces |
RELEASE.json, SHA256SUMS.txt, verify_release.py |
Release inventory and optional byte verification |
- Source: Voltline/VimeML, commit
54ace39b7fbbce4c362d55677c638ea2db7b83c6. - FP32 inference manifest:
9d9dad19c573c84c84e6a78060d1999a0a7e978c90a9bd78ca844da565a6ae74. - Core ML manifests: stateless
50a5524592117ff2c4775792213009e6aeb9f388524746fb8d8eda99fa248276, chunked KVc5bc759d4a224cf389d9ae28304df194e7e1b0ba3eb3945a94ecea9ddd82abc7, batched KV472d148a5c87948e5dad0d2ef3a352d34117715bf0a8fb6664f04d5bf2e8bf2a, batched KV v233c90e2a82e78e0ee52e7471f1a61c7e7e7b8dc9bdf233cc363702d5bc8147de, batched KV sequencec7b9e791bbebe58ebec384c92944803157a12d98af51459be11a61484c39bcc3, batched KV v37e4c01d2af202a725d2c6145e62eaeb0fc39ee74e65322ab0a7692f5d1eefd7b. - Tokenizer:
68bd37c45bcb018b32b2eef5b241561ab5b6dc1b77c1c3aadcc1b41c4bf47e5a. - Original training checkpoint:
44c423a1e7cb0e32161afc94d6167cabb439a3d25fd9de6d1fbb150a927542d4; the optimizer-bearing checkpoint is not distributed. The inference weights equal the checkpoint's model tensors exactly (the tied LM head is stored once).
These identities are SHA256 digests. Original model manifests remain unchanged. The internal codename of this model family in code and manifests is zh13 (tiny_gpt_zh13).
Architecture and tokenization
Pre-Norm RMSNorm/SwiGLU, bias-free Linear layers, learned absolute positions and causal attention. Vocabulary 12,288; context 128; width 384; 9 layers; 6 heads of dimension 64; SwiGLU width 960; dropout 0; RMSNorm epsilon 1e-5. Input embedding and LM head share one parameter.
SentencePiece Unigram, 12,288 pieces, trained on a sample of the cleaned corpus: identity normalization, whitespace preservation, byte fallback and add_dummy_prefix=false. PAD/UNK/BOS/EOS IDs are 0/1/2/3. IDs 4โ6 are reserved markers <CTX>, <READ>, <OUT> that the model never trains on; ID 7 is the newline. One 12K token covers 1.40 Chinese characters on average.
Quick start: downloaded files
Pythonโฅ3.11. Install the included requirements after selecting a suitable PyTorch build. FP32 inference does not invoke Apple runtime services.
python -m pip install -r source/requirements.txt
python verify_release.py
python infer.py --context "ๆไปฌๆๅคฉไธ่ตทๅป" --candidate "ๅ้ฅญ" --candidate "่ฟ่ฟ"
python infer.py --context "ๆไปฌๆๅคฉไธ่ตทๅป"
python infer.py --previous "ๅจๆซๆไปไนๅฎๆ๏ผ" --context "ๆๆณๅป"
The candidate command ranks supplied strings; it does not convert pinyin or retrieve dictionary candidates. Without candidates, infer.py runs the phrase-prediction decoder described below. The loader reads weights with torch.load(..., weights_only=True), restores the shared head, and checks the model configuration, parameter count and tokenizer contract. verify_release.py checks the complete inventory and sizes without running a model; --hashes additionally checks all file bytes against the recorded digests. Checksums are not a digital signature. Keep source/, the tokenizer and inference files together.
Candidate scoring
Encode BOS + context + candidate jointly. Find the common token prefix of the context and all candidate sequences, then sum full-vocabulary log-softmax over the remaining suffix. Do not append candidate-final EOS; ties keep original order. The score is the suffix log-likelihood under this tokenization. In deployment, librime with rime-ice produces the pinyin candidates and the model reorders them by pure LM score.
Phrase prediction (experimental) uses one beam decoder (source/src/vimeml/benchmarks/predict_zh.py): earlier turns are joined as separate BOS turn EOS sequences, the last pure-Chinese token of the typed text is taken back and re-spelled (token healing), beam width 16 over pure-Chinese tokens, and the top five 2โ6 character phrases that are not prefixes of one another are returned.
Training
Random initialization (seed 1591996171), one pass over fresh data, 844,730 updates, batch 512, BF16 with FP32 weights, AdamW (ฮฒ 0.9/0.95, weight decay 0.1, clip 1.0), cosine learning rate 6e-4โ6e-5 after 1,000 warmup updates, and a 30% prefix crop on eligible windows. Total exposure is 14,519,248,976 supervised next-token targets; training took 3.9 hours on a single GPU.
- Web text: 21 of the 370 train files of FineWeb2
cmn_Hani. Traditional-script documents were converted to Simplified Chinese with OpenCC; rule filters removed spam, gambling, pornographic and illegal-service pages and spam sites; flagged documents were kept only after review. A fixed hash-selected pool of 16,551,843 documents (21.28 billion characters) was seen once, split into paragraphs. - Dialogue: independent utterances from LCCC-base (16,326,323 training utterances, 209 million characters), sampled to 5% of supervised targets, which repeats the pool about four times.
Every paragraph and utterance is its own BOS text EOS sequence; the model never saw multi-turn context during training. The objective is ordinary next-token loss. This model is the release arm (12k-fresh-s1) of a preregistered eight-arm experiment comparing fresh data with repeated data and an 8K with a 12K tokenizer. None of the five contrasts was effective, so the preregistered defaults (fresh data, 12K) apply; the final checkpoint was used as registered.
FP32 validation BPC is 4.354450 on held-out web text and 5.690730 on held-out LCCC dialogue (total NLL including EOS divided by characters ร ln 2).
Evaluation
ime-standard-zh-v1-r1 reranks frozen librime + rime-ice candidate pools for real dialogue turns from CrossWOZ, RiSAWOZ and KdConv, using the left context; correct answers are not added to the pools. Labels are AI-reviewed, not native-speaker gold. The blind split was opened once for a preregistered confirmation, which passed; the baselines were scored afterwards with blind as a reference test.
| System | Parameters | Development Top-1 | Blind Top-1 | Blind Top-3 | Blind MRR |
|---|---|---|---|---|---|
| vimeml-tiny-zh-v1 | 20M | 93.6% (1,363/1,456) | 93.8% (1,355/1,445) | 97.2% | 0.954 |
| Qwen2.5-0.5B (same pools and scoring) | 494M | 91.3% | 91.8% | 96.1% | 0.940 |
| Qwen3-0.6B-Base (same pools and scoring) | 596M | 89.8% | 89.2% | 95.4% | 0.924 |
| rime-ice + wanxiang grammar model | โ | 79.0% | 82.2% | 83.7% | 0.829 |
| rime-ice | โ | 68.8% | 72.2% | 73.6% | 0.729 |
| luna_pinyin_simp | โ | 68.3% | 68.9% | 70.5% | 0.696 |
Against the rime-ice first candidate on blind: +329/โ19, exact McNemar p = 3.6e-74. All five paired comparisons on both splits remain significant after Holm correction; the closest system is Qwen2.5-0.5B (+28 on blind, p = 0.005). Abbreviated pinyin (initials only) is much harder: blind Top-1 35.0%, against 31.0% for Qwen2.5-0.5B and 15.7% for rime-ice + grammar; pool coverage limits it to about 41%.
Most of the advantage over Rime comes from using left context: with empty context the model scores 81.9% on development, close to rime-ice + grammar. Whether Qwen saw these public dialogue datasets during training is unknown.
Core ML variants
Weight-only symmetric INT8, block size 32; FP32 computation; CPU_ONLY; minimum iOS 18. No activation quantization or INT8 arithmetic is claimed. 47 matrices are compressed (token and position embeddings, fused QKV, attention projection and the three FFN matrices of every layer); RMSNorm weights, the causal mask and a converter-inserted zero head bias remain FP32. The stateless package is 22,792,958 bytes, versus 80,367,198 bytes uncompressed (about 71.6% smaller).
- Stateless:
input_idsINT32[1,T], 1โคTโค128, right PAD โlogitsFLOAT32[1,T,12288]. - Chunked KV (22,830,199 bytes):
input_idsINT32[1,T],positionINT32[1](current cache length, position+Tโค128),key_cache/value_cacheFLOAT32(9,1,6,128,64)โ logits for the new tokens plus updated caches. The caller owns cache snapshots and rollback. - Batched KV (22,762,761 bytes; added 2026-11-15): advances B independent hypotheses by one token in one call.
input_idsINT32[B,1],positionINT32[B](each row's cache length),key_cache/value_cacheFLOAT32(9,B,6,S,64)with B flexible 1..16 and S flexible 1..128 (passmax(position)+1slots) โlogitsFLOAT32[B,12288]for the new token plusnew_key/new_value(9,B,6,64), which the caller writes into slotposition[b]of row b. Rows never attend across each other. The prompt is prefilled once with the chunked package. It matches the stateless INT8 package within 4.8e-5 (atol/rtol 3e-4, all argmax equal), and a per-key-position beam search (beam 16) over it returns exactly the results of the same search run on the stateless INT8 package. Apple M3 Mac, CPU_ONLY, p50 per call: B=16 3.35 ms (S=16) / 3.76 ms (S=32); B=8 2.69 / 2.92 ms; B=1 1.64 ms. It was converted at VimeML commita8440c5d8c613e7e4964104089cff824e804637b(src/vimeml/deployment/zh_v1_batch.py);source/stays the original release's snapshot. - Batched KV v2 (22,774,484 bytes; added 2026-11-15; for beam search, superseded by batched KV v3): one package for both the prompt and the beam steps, so the chunked package is not needed.
input_idsINT32[B,T](B flexible 1..16, T flexible 1..128),positionINT32[B](each row's cache length, position+Tโค128),key_cache/value_cacheFLOAT32(9,B,6,S,64)with S flexible 1..128 and S โฅ max(position) (the caches are read only),allowedINT32[K](1โคKโค1024, one id list shared by all rows) โlogsumexpFLOAT32[B]over all 12,288 logits of the last new token,allowed_logitsFLOAT32[B,K](log P = allowed_logits โ logsumexp), andnew_key/new_value(9,B,6,T,64), which the caller writes into slotsposition[b]..of row b. Prompt: B=1, T = prompt length, position 0, S=1. It matches the stateless INT8 package within 3.0e-5 (atol/rtol 3e-4), and the per-key-position beam search (beam 16) over it returns exactly the results of the v1 batched package on all 125 fixture cases. Apple M3 Mac, CPU_ONLY, p50 per decode call: B=16 3.8โ4.4 ms and B=8 3.0โ3.5 ms at S up to 128; a 32-token prompt is one 3.8 ms call; search p50 79 ms per generation. It was converted at VimeML commit65c42fdf15e6167d52a752a2db06863c6842a8b4(src/vimeml/deployment/zh_v1_batch2.py). - Batched KV sequence (22,772,997 bytes; added 2026-11-16; for sentence scoring): the v2 graph, inputs and cache rules, with a sequence head. Extra input
targetsINT32[B,T](the token that followsinput_ids[b,t]) โtarget_logprobsFLOAT32[B,T], log P(targets[b,t]| row b throughinput_ids[b,t]) over all 12,288 logits, plusnew_key/new_value(9,B,6,T,64)as in v2. Scoring rule: prefill the shared prefix without its last token; then one call per 16 candidates withinput_ids= [last shared token] + candidate[:-1],targets= candidate,position= shared length โ 1, every row reading the same cache; a candidate's score is the sum of itstarget_logprobs. Rows of different lengths are right-padded and their padded outputs ignored. It matches PyTorch FP32 and the stateless INT8 package within 3.4e-5 (3e-4 per position, 1e-3 per sum), and on 8 fixture cases (39 candidates) it ranks the candidates exactly as PyTorch FP32. Apple M3 Mac, CPU_ONLY, p50: one B=16, T=16 call 17.5โ18.4 ms against 54.5โ62.2 ms for 16 batched KV v2 decode calls (3.1โ3.4ร); B=16, T=8 10.5โ11.2 ms. Memory grows with T because the head runs at every position: up to T=8 the footprint equals the v2 package's (~40 MiB over the stateless package); T=16 adds ~9 MiB retained (peak ~67 MiB), T=64 ~63 MiB. Cap T per call at 8 (or 16) and split longer continuations; the second call reads the first call's new keys and values from the cache. It does not replace the v2 package for beam search. It was converted at VimeML commitfcc17501841f355ce849619354b102987bd54e71(src/vimeml/deployment/zh_v1_batch2.py). - Batched KV v3 (23,139,825 bytes; added 2026-11-18; recommended for beam search and sentence scoring): one multifunction package (iOS 18:
MLModelConfiguration.functionName), three functions over one shared INT8 weight copy, so it replaces the batched KV v2 and sequence packages and can replace the stateless one. Every function takesprefix_lengthINT32[1]andprefix_key/prefix_valueFLOAT32(9,1,6,P,64): the prompt (BOS + context) passed once and read by every row, never copied into rows.logitsandscoredalso takeinput_idsINT32[B,T],positionINT32[B](tokens the row added after the prefix) and the row's ownkey_cache/value_cacheFLOAT32(9,B,6,S,64); the absolute position isprefix_length + position[b] + t.logitsโlogits[B,12288]of the last new token;scoredtakesallowedINT32[K]โlogsumexp[B]andallowed_logits[B,K]; both returnnew_key/new_value(9,B,6,T,64). A prompt call withP= 1,prefix_length= 0 and a one-slot empty row cache returns the prompt's keys and values, which become the prefix.scoretakesinput_ids,targetsINT32[B,T]and the prefix only โtarget_logprobs[B,T], with no row cache, no key/value outputs and no[B,T,12288]output; the scoring rule is the sequence package's with the prompt call's keys and values as the prefix. B 1..16,P + S + Tโค 128; masked prefix and cache slots must hold finite values. It matches PyTorch FP32 within 4.2e-5 and the stateless INT8 package within 4.15e-5 (3e-4 per value, 1e-3 per sum), and its beam search equals the batched KV v2 package on all 125 fixture cases. Apple M3 Mac, CPU_ONLY, p50 at B=16: decode 2.9โ3.3 ms (v2 3.0โ3.9 ms); scoring T=16 16.1โ17.1 ms (sequence package 17.2โ17.8 ms). Memory over the start (median of five fresh processes, compiled models): peak 18.5 / 31.4 / 56.8 MiB at scoring T = 16 / 32 / 64 against 40.5 / 70.2 / 150.0 MiB for stateless + v2 + sequence, ~4.5 MiB retained after work against 16โ57 MiB; mapped weights 23.1 MB alone. It was converted at VimeML commit586a04e827e54124e046f4b93295e4302ea78ed9(src/vimeml/deployment/zh_v1_batch3.py).
xcrun coremlcompiler compile coreml/ios18-int8-block32/model.mlpackage compiled-int8 --platform ios --deployment-target 18.0
Uncompressed conversions pass frozen logits alignment (maximum absolute error 3.9e-5 stateless, 4.6e-5 chunked over 22 checks). INT8 fails strict FP32 logits alignment (maximum 0.392, mean 0.046 at T=128; V4.2 Japanese reached 0.477) at unchanged atol/rtol 3e-4; padding and future-token invariants are exact. The chunked INT8 graph matches the stateless INT8 package within 4.2e-5 with identical rankings. On development, INT8 versus FP32 Top-1 is 1,364 vs 1,363 (+2/โ1) on full pinyin and 335 vs 337 (+0/โ2) on abbreviated pinyin; Top-3 is unchanged on both. The strict failure is recorded as INT8 quantization drift.
Performance evidence and limits
Apple M3 Mac, Core ML CPU_ONLY, measured interleaved with Japanese V4.2 in the same harness (median over five fresh processes):
| Metric | zh-v1 INT8 | V4.2 INT8 |
|---|---|---|
Stateless predict p50, T=1 / T=128 |
1.65 / 8.49 ms | 1.29 / 6.43 ms |
| Chunked KV prefill of 119 tokens / single-token decode, p50 | 9.33 / 2.71 ms | 7.18 / 2.05 ms |
| Model peak memory delta in the host process | 18.4 MiB | 13.4 MiB |
zh-v1 is about 1.3ร V4.2 in size and latency, in line with its larger vocabulary and two extra layers. INT8 reduces size, not CPU latency, because computation stays FP32. A simulator functional check (iOS 27.2) passed: resource identity, logits shape, native SentencePiece tokenization and joint suffix scores within 0.002 of Mac Core ML INT8.
iPhone 16 Pro Max, iOS 27.3, CPU_ONLY. Vime test host, four rounds alternating the two models, medians:
| Metric | zh-v1 INT8 | V4.2 INT8 |
|---|---|---|
Stateless predict p50, T=1 / T=128 |
1.70 / 6.82 ms | 1.25 / 4.90 ms |
| Load / reload | 195 / 7.4 ms | 155 / 5.9 ms |
| Memory delta after load and 50 predictions | 14.1 MiB | 11.3 MiB |
The chunked KV package loads in 236โ280 ms the first time. A single-token step takes 2.4โ2.9 ms p50 and a 128-token prefill 7.5โ7.6 ms p50. For 3โ6 short candidates the stateless joint scoring is faster (p50 about 9 ms versus 13.5โ14 ms with the KV package), so the keyboard uses the stateless package.
Inside the keyboard extension. Rime proposes the candidates. The model rescores at most the first 8 complete conversions after the sentence before the cursor; each place lower in Rime's order costs 1.0 log-probability. The candidate bar waits at most 60 ms for scores, then shows Rime's order. Learned phrases, partial conversions and excluded terms are never moved. Two five-minute runs of XCUITest taps on the installed keyboard (about 1.2 taps/s):
| Measured in the extension process | Model on | Model off |
|---|---|---|
| Key โ candidates tappable, mean / p95 / max | 26.1 / 45.0 / 49.7 ms | 20.3 / 26.1 / 27.4 ms |
| Model scoring, mean / p95 / max | 13.2 / 36.3 / 40.8 ms | โ |
| Physical footprint median / max; process peak | 20.0 / 29.5; 29.9 MiB | 24.3 / 25.3; 25.7 MiB |
| Model load (once) | 253 ms | โ |
p95 is the largest per-window p95 (windows of 256 calls). 221 of 222 scoring opportunities used the model's scores (the other showed Rime's order), and the model changed the order within the top 8 in 173 of the 221 lists. Scoring is several times slower in the extension than in the test host (per-keystroke scoring p50 1.9โ2.0 ms, p95 9.2โ9.8 ms over 300 scorings); the cause has not been identified. Chinese next-word prediction in the keyboard, behavior under memory pressure and energy use have not been measured.
Intended use and limitations
Simplified Chinese pinyin candidate reranking with left context, and experimental short phrase prediction, primarily for local input methods. This small non-instruction-tuned model can repeat, drift or invent text. It is not a standalone pinyin converter, general assistant or factual reference.
- Benchmark labels are AI-reviewed, and the data are dialogue turns rather than real keyboard traffic; candidate-pool coverage bounds achievable accuracy.
- The model was trained on single paragraphs and single utterances, so multi-turn context is outside its training distribution.
- Web and social-media text carry their own biases. Cleaning and review reduce inappropriate content; applications should keep their own output filtering.
- Public training overlap with evaluation data was checked by 16-character n-gram matching for the benchmark; semantic near-duplicates were not audited.
License and attribution
The weights, tokenizer, Core ML packages and included VimeML code are released under GNU GPL v3.0 or later with an additional permission under GPLv3 section 7 for conveyance through application distribution services such as the App Store. See LICENSE (GPLv3), LICENSE-EXCEPTION (copyright, additional permission and the source form of model weights) and NOTICE.md (training data attribution that applications bundling this model must show). Training sources and third-party dependencies retain their own terms; they are not relicensed.
Training data: FineWeb2 cmn_Hani (ODC-By 1.0, subject to the Common Crawl Terms of Use) and LCCC-base by thu-coai (Wang et al., A Large-Scale Chinese Short-Text Conversation Dataset, NLPCC 2020), whose repository and dataset card declare the MIT license and whose README states that the dataset and pretrained dialogue models are provided for research purposes. Evaluation sources: CrossWOZ (Apache-2.0), RiSAWOZ (MIT), KdConv (Apache-2.0) and CPED (Apache-2.0). This repository distributes the model and aggregate evaluation metrics only; it contains no corpus, dialogue or benchmark text. Optimizer/RNG state, user inputs, raw device traces, credentials and the separate Vime client are excluded.
- Downloads last month
- 64