VimeML Tiny Chinese GPT v1

A 20,036,736-parameter Simplified Chinese decoder-only language model for local contextual reranking of pinyin input-method candidates and short phrase prediction. This release contains the vimeml-tiny-zh-v1 step 844,730 FP32 inference bundle, its INT8 block32, FP32-compute Core ML stateless package, and a matching chunked KV-cache Core ML package.

Status: release for iOS deployment. The Vime input method on iOS uses the model to rerank pinyin candidates. The Core ML packages pass the Mac alignment gates. They have been measured on iPhone 16 Pro Max in a test host and inside the Vime keyboard extension, where the model reranks pinyin candidates (see Performance).

This is a custom PyTorch model, not a Transformers AutoModel integration or an instruction/chat model. SentencePiece, candidate retrieval (librime with rime-ice), candidate scoring and search run outside the Core ML graph. The architecture, loader and scoring code are included under source/; no trust_remote_code or training checkpoint is required.

Files and identity

Path Contents
inference/ Unique FP32 state_dict, config, tokenizer and provenance manifest; no optimizer
coreml/ios18-int8-block32/ Stateless Core ML .mlpackage and manifest
coreml/ios18-int8-block32-kvchunk/ Chunked KV-cache Core ML .mlpackage and manifest
coreml/ios18-int8-block32-kvbatch/ Batched single-step KV Core ML .mlpackage and manifest (pinyin beam search)
coreml/ios18-int8-block32-kvbatch-v2/ Batched KV v2 Core ML .mlpackage and manifest: multi-token prefill and decode in one package, scored head (pinyin beam search)
coreml/ios18-int8-block32-kvsequence/ Batched KV sequence-head Core ML .mlpackage and manifest: log-probability of a given token at every new position, so a batch of known continuations is scored in one call (sentence scoring)
coreml/ios18-int8-block32-kvbatch-v3/ Batched KV v3 Core ML multifunction .mlpackage and manifest: logits, scored and score functions over one weight copy, the prompt passed once as a shared prefix (pinyin beam search and sentence scoring; recommended)
source/ Frozen VimeML Python source, scripts, configurations and requirements
infer.py FP32 candidate-scoring and phrase-prediction wrapper using the zh-v1 bundle loader
evaluation/summary.json Aggregate quality, alignment and measurement results; no benchmark texts or traces
RELEASE.json, SHA256SUMS.txt, verify_release.py Release inventory and optional byte verification
  • Source: Voltline/VimeML, commit 54ace39b7fbbce4c362d55677c638ea2db7b83c6.
  • FP32 inference manifest: 9d9dad19c573c84c84e6a78060d1999a0a7e978c90a9bd78ca844da565a6ae74.
  • Core ML manifests: stateless 50a5524592117ff2c4775792213009e6aeb9f388524746fb8d8eda99fa248276, chunked KV c5bc759d4a224cf389d9ae28304df194e7e1b0ba3eb3945a94ecea9ddd82abc7, batched KV 472d148a5c87948e5dad0d2ef3a352d34117715bf0a8fb6664f04d5bf2e8bf2a, batched KV v2 33c90e2a82e78e0ee52e7471f1a61c7e7e7b8dc9bdf233cc363702d5bc8147de, batched KV sequence c7b9e791bbebe58ebec384c92944803157a12d98af51459be11a61484c39bcc3, batched KV v3 7e4c01d2af202a725d2c6145e62eaeb0fc39ee74e65322ab0a7692f5d1eefd7b.
  • Tokenizer: 68bd37c45bcb018b32b2eef5b241561ab5b6dc1b77c1c3aadcc1b41c4bf47e5a.
  • Original training checkpoint: 44c423a1e7cb0e32161afc94d6167cabb439a3d25fd9de6d1fbb150a927542d4; the optimizer-bearing checkpoint is not distributed. The inference weights equal the checkpoint's model tensors exactly (the tied LM head is stored once).

These identities are SHA256 digests. Original model manifests remain unchanged. The internal codename of this model family in code and manifests is zh13 (tiny_gpt_zh13).

Architecture and tokenization

Pre-Norm RMSNorm/SwiGLU, bias-free Linear layers, learned absolute positions and causal attention. Vocabulary 12,288; context 128; width 384; 9 layers; 6 heads of dimension 64; SwiGLU width 960; dropout 0; RMSNorm epsilon 1e-5. Input embedding and LM head share one parameter.

SentencePiece Unigram, 12,288 pieces, trained on a sample of the cleaned corpus: identity normalization, whitespace preservation, byte fallback and add_dummy_prefix=false. PAD/UNK/BOS/EOS IDs are 0/1/2/3. IDs 4โ€“6 are reserved markers <CTX>, <READ>, <OUT> that the model never trains on; ID 7 is the newline. One 12K token covers 1.40 Chinese characters on average.

Quick start: downloaded files

Pythonโ‰ฅ3.11. Install the included requirements after selecting a suitable PyTorch build. FP32 inference does not invoke Apple runtime services.

python -m pip install -r source/requirements.txt
python verify_release.py
python infer.py --context "ๆˆ‘ไปฌๆ˜Žๅคฉไธ€่ตทๅŽป" --candidate "ๅƒ้ฅญ" --candidate "่ฟŸ่ฟ”"
python infer.py --context "ๆˆ‘ไปฌๆ˜Žๅคฉไธ€่ตทๅŽป"
python infer.py --previous "ๅ‘จๆœซๆœ‰ไป€ไนˆๅฎ‰ๆŽ’๏ผŸ" --context "ๆˆ‘ๆƒณๅŽป"

The candidate command ranks supplied strings; it does not convert pinyin or retrieve dictionary candidates. Without candidates, infer.py runs the phrase-prediction decoder described below. The loader reads weights with torch.load(..., weights_only=True), restores the shared head, and checks the model configuration, parameter count and tokenizer contract. verify_release.py checks the complete inventory and sizes without running a model; --hashes additionally checks all file bytes against the recorded digests. Checksums are not a digital signature. Keep source/, the tokenizer and inference files together.

Candidate scoring

Encode BOS + context + candidate jointly. Find the common token prefix of the context and all candidate sequences, then sum full-vocabulary log-softmax over the remaining suffix. Do not append candidate-final EOS; ties keep original order. The score is the suffix log-likelihood under this tokenization. In deployment, librime with rime-ice produces the pinyin candidates and the model reorders them by pure LM score.

Phrase prediction (experimental) uses one beam decoder (source/src/vimeml/benchmarks/predict_zh.py): earlier turns are joined as separate BOS turn EOS sequences, the last pure-Chinese token of the typed text is taken back and re-spelled (token healing), beam width 16 over pure-Chinese tokens, and the top five 2โ€“6 character phrases that are not prefixes of one another are returned.

Training

Random initialization (seed 1591996171), one pass over fresh data, 844,730 updates, batch 512, BF16 with FP32 weights, AdamW (ฮฒ 0.9/0.95, weight decay 0.1, clip 1.0), cosine learning rate 6e-4โ†’6e-5 after 1,000 warmup updates, and a 30% prefix crop on eligible windows. Total exposure is 14,519,248,976 supervised next-token targets; training took 3.9 hours on a single GPU.

  • Web text: 21 of the 370 train files of FineWeb2 cmn_Hani. Traditional-script documents were converted to Simplified Chinese with OpenCC; rule filters removed spam, gambling, pornographic and illegal-service pages and spam sites; flagged documents were kept only after review. A fixed hash-selected pool of 16,551,843 documents (21.28 billion characters) was seen once, split into paragraphs.
  • Dialogue: independent utterances from LCCC-base (16,326,323 training utterances, 209 million characters), sampled to 5% of supervised targets, which repeats the pool about four times.

Every paragraph and utterance is its own BOS text EOS sequence; the model never saw multi-turn context during training. The objective is ordinary next-token loss. This model is the release arm (12k-fresh-s1) of a preregistered eight-arm experiment comparing fresh data with repeated data and an 8K with a 12K tokenizer. None of the five contrasts was effective, so the preregistered defaults (fresh data, 12K) apply; the final checkpoint was used as registered.

FP32 validation BPC is 4.354450 on held-out web text and 5.690730 on held-out LCCC dialogue (total NLL including EOS divided by characters ร— ln 2).

Evaluation

ime-standard-zh-v1-r1 reranks frozen librime + rime-ice candidate pools for real dialogue turns from CrossWOZ, RiSAWOZ and KdConv, using the left context; correct answers are not added to the pools. Labels are AI-reviewed, not native-speaker gold. The blind split was opened once for a preregistered confirmation, which passed; the baselines were scored afterwards with blind as a reference test.

System Parameters Development Top-1 Blind Top-1 Blind Top-3 Blind MRR
vimeml-tiny-zh-v1 20M 93.6% (1,363/1,456) 93.8% (1,355/1,445) 97.2% 0.954
Qwen2.5-0.5B (same pools and scoring) 494M 91.3% 91.8% 96.1% 0.940
Qwen3-0.6B-Base (same pools and scoring) 596M 89.8% 89.2% 95.4% 0.924
rime-ice + wanxiang grammar model โ€” 79.0% 82.2% 83.7% 0.829
rime-ice โ€” 68.8% 72.2% 73.6% 0.729
luna_pinyin_simp โ€” 68.3% 68.9% 70.5% 0.696

Against the rime-ice first candidate on blind: +329/โˆ’19, exact McNemar p = 3.6e-74. All five paired comparisons on both splits remain significant after Holm correction; the closest system is Qwen2.5-0.5B (+28 on blind, p = 0.005). Abbreviated pinyin (initials only) is much harder: blind Top-1 35.0%, against 31.0% for Qwen2.5-0.5B and 15.7% for rime-ice + grammar; pool coverage limits it to about 41%.

Most of the advantage over Rime comes from using left context: with empty context the model scores 81.9% on development, close to rime-ice + grammar. Whether Qwen saw these public dialogue datasets during training is unknown.

Core ML variants

Weight-only symmetric INT8, block size 32; FP32 computation; CPU_ONLY; minimum iOS 18. No activation quantization or INT8 arithmetic is claimed. 47 matrices are compressed (token and position embeddings, fused QKV, attention projection and the three FFN matrices of every layer); RMSNorm weights, the causal mask and a converter-inserted zero head bias remain FP32. The stateless package is 22,792,958 bytes, versus 80,367,198 bytes uncompressed (about 71.6% smaller).

  • Stateless: input_ids INT32 [1,T], 1โ‰คTโ‰ค128, right PAD โ†’ logits FLOAT32 [1,T,12288].
  • Chunked KV (22,830,199 bytes): input_ids INT32 [1,T], position INT32 [1] (current cache length, position+Tโ‰ค128), key_cache/value_cache FLOAT32 (9,1,6,128,64) โ†’ logits for the new tokens plus updated caches. The caller owns cache snapshots and rollback.
  • Batched KV (22,762,761 bytes; added 2026-11-15): advances B independent hypotheses by one token in one call. input_ids INT32 [B,1], position INT32 [B] (each row's cache length), key_cache/value_cache FLOAT32 (9,B,6,S,64) with B flexible 1..16 and S flexible 1..128 (pass max(position)+1 slots) โ†’ logits FLOAT32 [B,12288] for the new token plus new_key/new_value (9,B,6,64), which the caller writes into slot position[b] of row b. Rows never attend across each other. The prompt is prefilled once with the chunked package. It matches the stateless INT8 package within 4.8e-5 (atol/rtol 3e-4, all argmax equal), and a per-key-position beam search (beam 16) over it returns exactly the results of the same search run on the stateless INT8 package. Apple M3 Mac, CPU_ONLY, p50 per call: B=16 3.35 ms (S=16) / 3.76 ms (S=32); B=8 2.69 / 2.92 ms; B=1 1.64 ms. It was converted at VimeML commit a8440c5d8c613e7e4964104089cff824e804637b (src/vimeml/deployment/zh_v1_batch.py); source/ stays the original release's snapshot.
  • Batched KV v2 (22,774,484 bytes; added 2026-11-15; for beam search, superseded by batched KV v3): one package for both the prompt and the beam steps, so the chunked package is not needed. input_ids INT32 [B,T] (B flexible 1..16, T flexible 1..128), position INT32 [B] (each row's cache length, position+Tโ‰ค128), key_cache/value_cache FLOAT32 (9,B,6,S,64) with S flexible 1..128 and S โ‰ฅ max(position) (the caches are read only), allowed INT32 [K] (1โ‰คKโ‰ค1024, one id list shared by all rows) โ†’ logsumexp FLOAT32 [B] over all 12,288 logits of the last new token, allowed_logits FLOAT32 [B,K] (log P = allowed_logits โˆ’ logsumexp), and new_key/new_value (9,B,6,T,64), which the caller writes into slots position[b].. of row b. Prompt: B=1, T = prompt length, position 0, S=1. It matches the stateless INT8 package within 3.0e-5 (atol/rtol 3e-4), and the per-key-position beam search (beam 16) over it returns exactly the results of the v1 batched package on all 125 fixture cases. Apple M3 Mac, CPU_ONLY, p50 per decode call: B=16 3.8โ€“4.4 ms and B=8 3.0โ€“3.5 ms at S up to 128; a 32-token prompt is one 3.8 ms call; search p50 79 ms per generation. It was converted at VimeML commit 65c42fdf15e6167d52a752a2db06863c6842a8b4 (src/vimeml/deployment/zh_v1_batch2.py).
  • Batched KV sequence (22,772,997 bytes; added 2026-11-16; for sentence scoring): the v2 graph, inputs and cache rules, with a sequence head. Extra input targets INT32 [B,T] (the token that follows input_ids[b,t]) โ†’ target_logprobs FLOAT32 [B,T], log P(targets[b,t] | row b through input_ids[b,t]) over all 12,288 logits, plus new_key/new_value (9,B,6,T,64) as in v2. Scoring rule: prefill the shared prefix without its last token; then one call per 16 candidates with input_ids = [last shared token] + candidate[:-1], targets = candidate, position = shared length โˆ’ 1, every row reading the same cache; a candidate's score is the sum of its target_logprobs. Rows of different lengths are right-padded and their padded outputs ignored. It matches PyTorch FP32 and the stateless INT8 package within 3.4e-5 (3e-4 per position, 1e-3 per sum), and on 8 fixture cases (39 candidates) it ranks the candidates exactly as PyTorch FP32. Apple M3 Mac, CPU_ONLY, p50: one B=16, T=16 call 17.5โ€“18.4 ms against 54.5โ€“62.2 ms for 16 batched KV v2 decode calls (3.1โ€“3.4ร—); B=16, T=8 10.5โ€“11.2 ms. Memory grows with T because the head runs at every position: up to T=8 the footprint equals the v2 package's (~40 MiB over the stateless package); T=16 adds ~9 MiB retained (peak ~67 MiB), T=64 ~63 MiB. Cap T per call at 8 (or 16) and split longer continuations; the second call reads the first call's new keys and values from the cache. It does not replace the v2 package for beam search. It was converted at VimeML commit fcc17501841f355ce849619354b102987bd54e71 (src/vimeml/deployment/zh_v1_batch2.py).
  • Batched KV v3 (23,139,825 bytes; added 2026-11-18; recommended for beam search and sentence scoring): one multifunction package (iOS 18: MLModelConfiguration.functionName), three functions over one shared INT8 weight copy, so it replaces the batched KV v2 and sequence packages and can replace the stateless one. Every function takes prefix_length INT32 [1] and prefix_key/prefix_value FLOAT32 (9,1,6,P,64): the prompt (BOS + context) passed once and read by every row, never copied into rows. logits and scored also take input_ids INT32 [B,T], position INT32 [B] (tokens the row added after the prefix) and the row's own key_cache/value_cache FLOAT32 (9,B,6,S,64); the absolute position is prefix_length + position[b] + t. logits โ†’ logits [B,12288] of the last new token; scored takes allowed INT32 [K] โ†’ logsumexp [B] and allowed_logits [B,K]; both return new_key/new_value (9,B,6,T,64). A prompt call with P = 1, prefix_length = 0 and a one-slot empty row cache returns the prompt's keys and values, which become the prefix. score takes input_ids, targets INT32 [B,T] and the prefix only โ†’ target_logprobs [B,T], with no row cache, no key/value outputs and no [B,T,12288] output; the scoring rule is the sequence package's with the prompt call's keys and values as the prefix. B 1..16, P + S + T โ‰ค 128; masked prefix and cache slots must hold finite values. It matches PyTorch FP32 within 4.2e-5 and the stateless INT8 package within 4.15e-5 (3e-4 per value, 1e-3 per sum), and its beam search equals the batched KV v2 package on all 125 fixture cases. Apple M3 Mac, CPU_ONLY, p50 at B=16: decode 2.9โ€“3.3 ms (v2 3.0โ€“3.9 ms); scoring T=16 16.1โ€“17.1 ms (sequence package 17.2โ€“17.8 ms). Memory over the start (median of five fresh processes, compiled models): peak 18.5 / 31.4 / 56.8 MiB at scoring T = 16 / 32 / 64 against 40.5 / 70.2 / 150.0 MiB for stateless + v2 + sequence, ~4.5 MiB retained after work against 16โ€“57 MiB; mapped weights 23.1 MB alone. It was converted at VimeML commit 586a04e827e54124e046f4b93295e4302ea78ed9 (src/vimeml/deployment/zh_v1_batch3.py).
xcrun coremlcompiler compile coreml/ios18-int8-block32/model.mlpackage compiled-int8 --platform ios --deployment-target 18.0

Uncompressed conversions pass frozen logits alignment (maximum absolute error 3.9e-5 stateless, 4.6e-5 chunked over 22 checks). INT8 fails strict FP32 logits alignment (maximum 0.392, mean 0.046 at T=128; V4.2 Japanese reached 0.477) at unchanged atol/rtol 3e-4; padding and future-token invariants are exact. The chunked INT8 graph matches the stateless INT8 package within 4.2e-5 with identical rankings. On development, INT8 versus FP32 Top-1 is 1,364 vs 1,363 (+2/โˆ’1) on full pinyin and 335 vs 337 (+0/โˆ’2) on abbreviated pinyin; Top-3 is unchanged on both. The strict failure is recorded as INT8 quantization drift.

Performance evidence and limits

Apple M3 Mac, Core ML CPU_ONLY, measured interleaved with Japanese V4.2 in the same harness (median over five fresh processes):

Metric zh-v1 INT8 V4.2 INT8
Stateless predict p50, T=1 / T=128 1.65 / 8.49 ms 1.29 / 6.43 ms
Chunked KV prefill of 119 tokens / single-token decode, p50 9.33 / 2.71 ms 7.18 / 2.05 ms
Model peak memory delta in the host process 18.4 MiB 13.4 MiB

zh-v1 is about 1.3ร— V4.2 in size and latency, in line with its larger vocabulary and two extra layers. INT8 reduces size, not CPU latency, because computation stays FP32. A simulator functional check (iOS 27.2) passed: resource identity, logits shape, native SentencePiece tokenization and joint suffix scores within 0.002 of Mac Core ML INT8.

iPhone 16 Pro Max, iOS 27.3, CPU_ONLY. Vime test host, four rounds alternating the two models, medians:

Metric zh-v1 INT8 V4.2 INT8
Stateless predict p50, T=1 / T=128 1.70 / 6.82 ms 1.25 / 4.90 ms
Load / reload 195 / 7.4 ms 155 / 5.9 ms
Memory delta after load and 50 predictions 14.1 MiB 11.3 MiB

The chunked KV package loads in 236โ€“280 ms the first time. A single-token step takes 2.4โ€“2.9 ms p50 and a 128-token prefill 7.5โ€“7.6 ms p50. For 3โ€“6 short candidates the stateless joint scoring is faster (p50 about 9 ms versus 13.5โ€“14 ms with the KV package), so the keyboard uses the stateless package.

Inside the keyboard extension. Rime proposes the candidates. The model rescores at most the first 8 complete conversions after the sentence before the cursor; each place lower in Rime's order costs 1.0 log-probability. The candidate bar waits at most 60 ms for scores, then shows Rime's order. Learned phrases, partial conversions and excluded terms are never moved. Two five-minute runs of XCUITest taps on the installed keyboard (about 1.2 taps/s):

Measured in the extension process Model on Model off
Key โ†’ candidates tappable, mean / p95 / max 26.1 / 45.0 / 49.7 ms 20.3 / 26.1 / 27.4 ms
Model scoring, mean / p95 / max 13.2 / 36.3 / 40.8 ms โ€”
Physical footprint median / max; process peak 20.0 / 29.5; 29.9 MiB 24.3 / 25.3; 25.7 MiB
Model load (once) 253 ms โ€”

p95 is the largest per-window p95 (windows of 256 calls). 221 of 222 scoring opportunities used the model's scores (the other showed Rime's order), and the model changed the order within the top 8 in 173 of the 221 lists. Scoring is several times slower in the extension than in the test host (per-keystroke scoring p50 1.9โ€“2.0 ms, p95 9.2โ€“9.8 ms over 300 scorings); the cause has not been identified. Chinese next-word prediction in the keyboard, behavior under memory pressure and energy use have not been measured.

Intended use and limitations

Simplified Chinese pinyin candidate reranking with left context, and experimental short phrase prediction, primarily for local input methods. This small non-instruction-tuned model can repeat, drift or invent text. It is not a standalone pinyin converter, general assistant or factual reference.

  • Benchmark labels are AI-reviewed, and the data are dialogue turns rather than real keyboard traffic; candidate-pool coverage bounds achievable accuracy.
  • The model was trained on single paragraphs and single utterances, so multi-turn context is outside its training distribution.
  • Web and social-media text carry their own biases. Cleaning and review reduce inappropriate content; applications should keep their own output filtering.
  • Public training overlap with evaluation data was checked by 16-character n-gram matching for the benchmark; semantic near-duplicates were not audited.

License and attribution

The weights, tokenizer, Core ML packages and included VimeML code are released under GNU GPL v3.0 or later with an additional permission under GPLv3 section 7 for conveyance through application distribution services such as the App Store. See LICENSE (GPLv3), LICENSE-EXCEPTION (copyright, additional permission and the source form of model weights) and NOTICE.md (training data attribution that applications bundling this model must show). Training sources and third-party dependencies retain their own terms; they are not relicensed.

Training data: FineWeb2 cmn_Hani (ODC-By 1.0, subject to the Common Crawl Terms of Use) and LCCC-base by thu-coai (Wang et al., A Large-Scale Chinese Short-Text Conversation Dataset, NLPCC 2020), whose repository and dataset card declare the MIT license and whose README states that the dataset and pretrained dialogue models are provided for research purposes. Evaluation sources: CrossWOZ (Apache-2.0), RiSAWOZ (MIT), KdConv (Apache-2.0) and CPED (Apache-2.0). This repository distributes the model and aggregate evaluation metrics only; it contains no corpus, dialogue or benchmark text. Optimizer/RNG state, user inputs, raw device traces, credentials and the separate Vime client are excluded.

Downloads last month
64
Inference Providers NEW
This model isn't deployed by any Inference Provider. ๐Ÿ™‹ Ask for provider support