Wai Chung's picture

Wai Chung PRO

Ternii

AI & ML interests

None yet

Recent Activity

repliedto their post 19 minutes ago
llama.cpp-tr r4 is out: our trit-quantised models now run on CUDA as well as Metal. New in r4: - A Linux arm64 build with CUDA for the DGX Spark, alongside the macOS build. On a DGX Spark, prompt processing at 2,048 tokens went from 1,387 to 1,924 tokens/s on Qwen3.6-35B-A3B (×1.39) and from 607 to 806 on Qwen3.8-Flash-Next (×1.33), compared with the build's plain CUDA kernels. - On the Mac, a Metal kernel for 2–8 tokens at a time (several users decoding together). With 2–8 users, total decode is 8–24 % faster; with one user it is level (no speed-up). Answers are identical to r3 on our 60-prompt check. (Corrected 22 Nov: this line first said the kernel speeds up speculative-decoding checks; with one user those are ~65-token passes that use a different kernel.) - x86 (AVX2) support in the source; build it yourself with `apply.sh`. No x86 binary yet. Accuracy: the CUDA tensor-core prompt path changes outputs slightly. We measured it against the path switched off (`GGML_CUDA_TQ_MM=0`) and found no significant difference: Qwen3.6-35B-A3B GSM8K 90.83 vs 90.60, HumanEval+ 91.46 vs 92.68, MBPP+ 91.53 vs 91.27; Qwen3.8-Flash-Next GSM8K 92.80 vs 93.25 (n = 1,319). Caveats: CUDA is tested on a DGX Spark (GB10) only. The one-line Mac installers stay on r3. The builds need our patched llama.cpp; LM Studio and Ollama can't load these models. Release: https://github.com/ternii-ai/tritq-vllm/releases/tag/llama.cpp-tr-b11721-r4 Models: https://huggingface.co/Ternii/Qwen3.6-35B-A3B-TR2.25-G-GGUF · https://huggingface.co/Ternii/Qwen3.8-Flash-Next-TR2.31-G-GGUF
updated a model 1 day ago
Ternii/Qwen3.8-Flash-Next-TR2.31-G-GGUF
updated a model 1 day ago
Ternii/Qwen3.6-35B-A3B-TR2.25-G-GGUF
View all activity

Organizations

None yet