Instructions to use HelloSun/lingtiny30vq3 with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Notebooks
- Google Colab
- Kaggle
- Local Apps Settings
- llama.cpp
How to use HelloSun/lingtiny30vq3 with llama.cpp:
Install (macOS, Linux)
curl -LsSf https://llama.app/install.sh | sh # Start a local OpenAI-compatible server with a web UI: llama serve -hf HelloSun/lingtiny30vq3 # Run inference directly in the terminal: llama cli -hf HelloSun/lingtiny30vq3
Install from WinGet (Windows)
winget install llama.cpp # Start a local OpenAI-compatible server with a web UI: llama serve -hf HelloSun/lingtiny30vq3 # Run inference directly in the terminal: llama cli -hf HelloSun/lingtiny30vq3
Build from source code
git clone https://github.com/ggml-org/llama.cpp.git cd llama.cpp cmake -B build cmake --build build -j --target llama-app # Start a local OpenAI-compatible server with a web UI: ./build/bin/llama serve -hf HelloSun/lingtiny30vq3 # Run inference directly in the terminal: ./build/bin/llama cli -hf HelloSun/lingtiny30vq3
Use Docker
docker model run hf.co/HelloSun/lingtiny30vq3
- Llama new
- LM Studio
- Jan
- Ollama
How to use HelloSun/lingtiny30vq3 with Ollama:
ollama run hf.co/HelloSun/lingtiny30vq3
- Unsloth Desktop
- Pi
How to use HelloSun/lingtiny30vq3 with Pi:
Start the llama.cpp server
# Install llama.cpp: brew install llama.cpp # Start a local OpenAI-compatible server: llama serve -hf HelloSun/lingtiny30vq3
Configure the model in Pi
# Install Pi: npm install -g @earendil-works/pi-coding-agent # Add to ~/.pi/agent/models.json: { "providers": { "llama-cpp": { "baseUrl": "http://localhost:9931/v1", "api": "openai-completions", "apiKey": "none", "models": [ { "id": "HelloSun/lingtiny30vq3" } ] } } }Run Pi
# Start Pi in your project directory: pi
- Docker Model Runner
How to use HelloSun/lingtiny30vq3 with Docker Model Runner:
docker model run hf.co/HelloSun/lingtiny30vq3
- Lemonade
How to use HelloSun/lingtiny30vq3 with Lemonade:
Pull the model
# Download Lemonade from https://lemonade-server.ai/ lemonade pull HelloSun/lingtiny30vq3
Run and chat with the model
lemonade run user.lingtiny30vq3-{{QUANT_TAG}}List all available models
lemonade list
- Hermes Agent
How to use HelloSun/lingtiny30vq3 with Hermes Agent:
Start the llama.cpp server
# Install llama.cpp: brew install llama.cpp # Start a local OpenAI-compatible server: llama serve -hf HelloSun/lingtiny30vq3
Configure Hermes
# Install Hermes: curl -fsSL https://hermes-agent.nousresearch.com/install.sh | bash hermes setup # Point Hermes at the local server: hermes config set model.provider custom hermes config set model.base_url http://127.0.0.1:9931/v1 hermes config set model.default HelloSun/lingtiny30vq3
Run Hermes
hermes
- Atomic Chat
- OpenClaw
How to use HelloSun/lingtiny30vq3 with OpenClaw:
Start the llama.cpp server
# Install llama.cpp: brew install llama.cpp # Start a local OpenAI-compatible server: llama serve -hf HelloSun/lingtiny30vq3
Configure OpenClaw
# Install OpenClaw: npm install -g openclaw@latest # Register the local server and set it as the default model: openclaw onboard --non-interactive --mode local \ --auth-choice custom-api-key \ --custom-base-url http://127.0.0.1:9931/v1 \ --custom-model-id "HelloSun/lingtiny30vq3" \ --custom-provider-id llama-cpp \ --custom-compatibility openai \ --custom-text-input \ --accept-risk \ --skip-health
Run OpenClaw
openclaw agent --local --agent main --message "Hello from Hugging Face"
Install from WinGet (Windows)
winget install llama.cpp
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf HelloSun/lingtiny30vq3# Run inference directly in the terminal:
llama cli -hf HelloSun/lingtiny30vq3Build from source code
git clone https://github.com/ggml-org/llama.cpp.git
cd llama.cpp
cmake -B build
cmake --build build -j --target llama-app# Start a local OpenAI-compatible server with a web UI:
./build/bin/llama serve -hf HelloSun/lingtiny30vq3# Run inference directly in the terminal:
./build/bin/llama cli -hf HelloSun/lingtiny30vq3Use Docker
docker model run hf.co/HelloSun/lingtiny30vq3Ling-3.0-tiny LUT-VQ3-64(VQ3_0,專家張量 3.00 bpw)
inclusionAI/Ling-3.0-tiny(7.89B MoE,BF16 master,MIT)之 LUT-VQ3-64 向量量化版本。
46 個專家張量(ffn_gate_exps/ffn_up_exps)以 128 條目 3D int8 碼本 + 3-bit 子區塊尺度
量化(3.00 bpw),推論端為 llama.cpp 補丁新增的 GGML_TYPE_VQ3_0 核心
(AVX-512 VBMI VPERMI2B LUT 快速路徑)。本 repo 版本針對 2×40C Xeon(雙 NUMA、
完整 AVX-512+AMX)調校,含 128K 上下文實測數據。
- 模型:
Ling-3.0-tiny-VQ3.gguf(4.66 GB / 4.34 GiB,n_tensors=526,有效率 4.73 BPW) - 配方:gate/up_exps → VQ3_0;down_exps → Q6_K;其餘 → Q8_0
一鍵編譯 + 下載 + 實驗
git clone https://huggingface.co/HelloSun/lingtimy30vq3 && cd lingtimy30vq3
bash build_and_run.sh --all # 編譯→測試→下載模型→PPL/bench→128k 實驗
本機最佳化設定(2×40C Xeon 實測)
| 設定 | pp512 | tg128 | 結論 |
|---|---|---|---|
-t 32 |
268.9 | 38.0 | decode 最快(比 t64 快 +15%) |
-t 48 |
344.5 | 35.2 | 均衡 |
-t 64 |
353.3 | 33.8 | prefill 最快 |
-t 80 |
68.3 | ~0 | ❌ 跨 NUMA 抖動,避免 |
--numa distribute |
全線下降 | t80 崩至 0.9 | ❌ 本機不要用 |
**推薦:-t 32 -tb 64**(decode 32 執行緒 / prefill-batch 64 執行緒,兩者兼得)。
原始數據:results/tuning.tsv。
短上下文基準(ctx 512–4096)
| 模型 | 大小 | PPL ↓ | KLD vs BF16 ↓ | pp512 | tg128 | 尖峰 RSS |
|---|---|---|---|---|---|---|
| BF16(master) | 14.71 GiB | 13.1045 | — | 425 t/s | 26.9 t/s | 15.70 GiB |
| Q8_0 | 7.82 GiB | 13.0718 | 0.012 | 504 t/s | 33.0 t/s | 9.32 GiB |
| base-Q8dQ6 | 7.30 GiB | 13.1406 | 0.015 | 458 t/s | 34.2 t/s | 8.80 GiB |
| Q4_K_M | 4.49 GiB | 13.4217 | 0.133 | 424 t/s | 33.7 t/s | 8.74 GiB |
| VQ3(本 repo) | 4.34 GiB | 13.8465 | 0.255 | 369 t/s | 32.8 t/s | 5.84 GiB |
ctx 128K 實測(-c 131072,prompt ≈131k tokens,-t 32 -tb 64)
| 模型 | ub | prefill(131k tokens) | decode(滿 128k KV) | 尖峰 RSS |
|---|---|---|---|---|
| VQ3 | 512 | 40.7 t/s | 6.5 t/s | 6.10 GiB |
| VQ3(最佳化) | 1024 | 71.2 t/s(+75%) | 6.0 t/s | 6.27 GiB |
| VQ3 | 2048 | 71.8 t/s | 6.1 t/s | 6.62 GiB |
| Q8_0 | 512 | 41.9 t/s | 6.2 t/s | 9.59 GiB |
| Q4_K_M | 512 | 41.4 t/s | 6.5 t/s | 9.18 GiB |
| base-Q8dQ6 | 512 | 41.6 t/s | 6.0 t/s | 9.06 GiB |
| BF16 | 512 | 39.7 t/s | 5.8 t/s | 15.95 GiB |
- 128k prefill 主要受 ubatch 影響:
-ub 1024把 VQ3 prefill 從 40.7 提到 71.2 t/s (ub2048 無進一步增益且 RSS +350 MiB → ub 1024 為最佳點) - 128k decode 受 KV 讀取支配:MLA 壓縮 KV 每 token 576×2 B ×6 層,滿 context 每解碼一 token 需讀 ~0.9 GiB KV + 權重 → 頻寬受限;此時 VQ3 權重讀取量最小, decode 反超 Q8_0(6.5 vs 6.2 t/s)
- 明細:run_vq3.txt · ub1024 · 各模型 RSS
使用
llama-cli -m Ling-3.0-tiny-VQ3.gguf -p "你好" -n 256 -t 32 -tb 64 -ngl 0
llama-server -m Ling-3.0-tiny-VQ3.gguf -t 32 -tb 64 -ngl 0 --host 0.0.0.0 --port 8080
# 128k 長文(本機最佳化:ub 1024)
llama-cli -m Ling-3.0-tiny-VQ3.gguf -f long.txt -c 131072 -n 256 -t 32 -tb 64 -ub 1024 -ngl 0
重新量化(可重現)
llama-quantize --tensor-type "ffn_down_exps=Q6_K" \
Ling-3.0-tiny-bf16.gguf Ling-3.0-tiny-base-Q8dQ6.gguf Q8_0
gcc -O3 -march=native -mavx512f -mf16c -fopenmp -shared -fPIC \
vq3/vq3_enc_lib.c -lm -o vq3/libvq3_enc.so
python3 scripts/build_ling_vq3.py \
--base-gguf Ling-3.0-tiny-base-Q8dQ6.gguf \
--bf16-gguf Ling-3.0-tiny-bf16.gguf --output Ling-3.0-tiny-VQ3.gguf
連結
- 基礎模型:inclusionAI/Ling-3.0-tiny(MIT)
- 上游 llama.cpp:ggml-org/llama.cpp @
61dbf492 - VQ3 格式:synchopate/llama-modchip(LUT-VQ3-64)
- 舊版 repo(vp3 拼字):HelloSun/lingtimy30vp3
注意事項
- 補丁含 AMX 修復:3D MLA 權重(
attn_k_b/attn_v_b)被 AMX buffer 認領後, bailingmoe3 以 permuted view 之 MUL_MAT 消費會崩潰;已於ggml-cpu/amx/amx.cpp加ggml_n_dims(src0)==2檢查(BF16/Q8_0/Q4_K_M 同樣需要)。 - 核心正確性:SQNR 13.45 dB、cos 0.977 @ 3.00 bpw(
tests/test_vq3_port.c)。
授權
MIT(同基礎模型)。
- Downloads last month
- 135
We're not able to determine the quantization variants.
Model tree for HelloSun/lingtiny30vq3
Base model
inclusionAI/Ling-3.0-tiny
Install (macOS, Linux)
# Start a local OpenAI-compatible server with a web UI: llama serve -hf HelloSun/lingtiny30vq3# Run inference directly in the terminal: llama cli -hf HelloSun/lingtiny30vq3