Post
479
llama.cpp-tr r4 is out: our trit-quantised models now run on CUDA as well as Metal.
New in r4:
- A Linux arm64 build with CUDA for the DGX Spark, alongside the macOS build. On a DGX Spark, prompt processing at 2,048 tokens went from 1,387 to 1,924 tokens/s on Qwen3.6-35B-A3B (×1.39) and from 607 to 806 on Qwen3.8-Flash-Next (×1.33), compared with the build's plain CUDA kernels.
- On the Mac, a Metal kernel for 2–8 tokens at a time (several users decoding together). With 2–8 users, total decode is 8–24 % faster; with one user it is level (no speed-up). Answers are identical to r3 on our 60-prompt check. (Corrected 22 Nov: this line first said the kernel speeds up speculative-decoding checks; with one user those are ~65-token passes that use a different kernel.)
- x86 (AVX2) support in the source; build it yourself with
Accuracy: the CUDA tensor-core prompt path changes outputs slightly. We measured it against the path switched off (
Caveats: CUDA is tested on a DGX Spark (GB10) only. The one-line Mac installers stay on r3. The builds need our patched llama.cpp; LM Studio and Ollama can't load these models.
Release: https://github.com/ternii-ai/tritq-vllm/releases/tag/llama.cpp-tr-b11721-r4
Models: Ternii/Qwen3.6-35B-A3B-TR2.25-G-GGUF · Ternii/Qwen3.8-Flash-Next-TR2.31-G-GGUF
New in r4:
- A Linux arm64 build with CUDA for the DGX Spark, alongside the macOS build. On a DGX Spark, prompt processing at 2,048 tokens went from 1,387 to 1,924 tokens/s on Qwen3.6-35B-A3B (×1.39) and from 607 to 806 on Qwen3.8-Flash-Next (×1.33), compared with the build's plain CUDA kernels.
- On the Mac, a Metal kernel for 2–8 tokens at a time (several users decoding together). With 2–8 users, total decode is 8–24 % faster; with one user it is level (no speed-up). Answers are identical to r3 on our 60-prompt check. (Corrected 22 Nov: this line first said the kernel speeds up speculative-decoding checks; with one user those are ~65-token passes that use a different kernel.)
- x86 (AVX2) support in the source; build it yourself with
apply.sh. No x86 binary yet.Accuracy: the CUDA tensor-core prompt path changes outputs slightly. We measured it against the path switched off (
GGML_CUDA_TQ_MM=0) and found no significant difference: Qwen3.6-35B-A3B GSM8K 90.83 vs 90.60, HumanEval+ 91.46 vs 92.68, MBPP+ 91.53 vs 91.27; Qwen3.8-Flash-Next GSM8K 92.80 vs 93.25 (n = 1,319).Caveats: CUDA is tested on a DGX Spark (GB10) only. The one-line Mac installers stay on r3. The builds need our patched llama.cpp; LM Studio and Ollama can't load these models.
Release: https://github.com/ternii-ai/tritq-vllm/releases/tag/llama.cpp-tr-b11721-r4
Models: Ternii/Qwen3.6-35B-A3B-TR2.25-G-GGUF · Ternii/Qwen3.8-Flash-Next-TR2.31-G-GGUF