AxiomicLabs/Tiny_Theory_of_Mind
Viewer โข Updated โข 2k โข 241 โข 21
Saeed
MLDataScientist
AI & ML interests
None yet
Recent Activity
liked a dataset about 23 hours ago
AxiomicLabs/Tiny_Theory_of_Mind liked a model 7 days ago
ISTA-DASLab/Qwen3.8-27B-GSQ-RCO-GGUF liked a model 11 days ago
byteshape/Qwen3.8-27B-GGUFOrganizations
None yet
liked a
dataset about 23 hours ago
liked a
model 7 days ago
liked a
model 11 days ago
liked a
model 13 days ago
liked a
model 16 days ago
upvoted a collection 22 days ago
liked 2
models about 1 month ago
liked a
model about 2 months ago
liked a
model 3 months ago
Does it work with expert offloading to CPU?
1
#2 opened 3 months ago
by MLDataScientist
Which llama.cpp version?
3
#2 opened 3 months ago
by MLDataScientist
We need some Air or at least some Flash
โค๏ธ 183
43
#3 opened 3 months ago
by 
jacek2024
liked a
model 5 months ago
Hi uzbek datasets
3
#2 opened about 1 year ago
by 
ZiyoVer
4-bit
39
#1 opened over 1 year ago
by 
prashanthellina
Where is UD_ Q2_K_XL?
2
#21 opened over 1 year ago
by MLDataScientist
reacted to wolfram's post with ๐ฅ over 1 year ago
Post
8511
Finally finished my extensive **Qwen 3 evaluations** across a range of formats and quantisations, focusing on **MMLU-Pro** (Computer Science).
A few take-aways stood out - especially for those interested in local deployment and performance trade-offs:
1๏ธโฃ **Qwen3-235B-A22B** (via Fireworks API) tops the table at **83.66%** with ~55 tok/s.
2๏ธโฃ But the **30B-A3B Unsloth** quant delivered **82.20%** while running locally at ~45 tok/s and with zero API spend.
3๏ธโฃ The same Unsloth build is ~5x faster than Qwen's **Qwen3-32B**, which scores **82.20%** as well yet crawls at <10 tok/s.
4๏ธโฃ On Apple silicon, the **30B MLX** port hits **79.51%** while sustaining ~64 tok/s - arguably today's best speed/quality trade-off for Mac setups.
5๏ธโฃ The **0.6B** micro-model races above 180 tok/s but tops out at **37.56%** - that's why it's not even on the graph (50 % performance cut-off).
All local runs were done with LM Studio on an M4 MacBook Pro, using Qwen's official recommended settings.
**Conclusion:** Quantised 30B models now get you ~98 % of frontier-class accuracy - at a fraction of the latency, cost, and energy. For most local RAG or agent workloads, they're not just good enough - they're the new default.
Well done, Qwen - you really whipped the llama's ass! And to OpenAI: for your upcoming open model, please make it MoE, with toggleable reasoning, and release it in many sizes. *This* is the future!
A few take-aways stood out - especially for those interested in local deployment and performance trade-offs:
1๏ธโฃ **Qwen3-235B-A22B** (via Fireworks API) tops the table at **83.66%** with ~55 tok/s.
2๏ธโฃ But the **30B-A3B Unsloth** quant delivered **82.20%** while running locally at ~45 tok/s and with zero API spend.
3๏ธโฃ The same Unsloth build is ~5x faster than Qwen's **Qwen3-32B**, which scores **82.20%** as well yet crawls at <10 tok/s.
4๏ธโฃ On Apple silicon, the **30B MLX** port hits **79.51%** while sustaining ~64 tok/s - arguably today's best speed/quality trade-off for Mac setups.
5๏ธโฃ The **0.6B** micro-model races above 180 tok/s but tops out at **37.56%** - that's why it's not even on the graph (50 % performance cut-off).
All local runs were done with LM Studio on an M4 MacBook Pro, using Qwen's official recommended settings.
**Conclusion:** Quantised 30B models now get you ~98 % of frontier-class accuracy - at a fraction of the latency, cost, and energy. For most local RAG or agent workloads, they're not just good enough - they're the new default.
Well done, Qwen - you really whipped the llama's ass! And to OpenAI: for your upcoming open model, please make it MoE, with toggleable reasoning, and release it in many sizes. *This* is the future!
- 4 replies
UD quants missing some files
โ 3
6
#2 opened over 1 year ago
by MLDataScientist
UD quants missing some files
๐ 1
1
#2 opened over 1 year ago
by MLDataScientist