Hoglet-33/3.33BT-Pre-Training-Mix-Three-Of-Three
Viewer • Updated • 3.9M • 68
Hoglet (Ash) PRO
Hoglet-33
AI & ML interests
Open source AI, datasets, parameter efficiency, SLMs, AI for the betterment of humanity. Contact at ash@basicallyai.co
Recent Activity
new activity 31 minutes ago
Project-Prism/REFRACT-AI-LABS-INVITATIONS:Hello! repliedto Banaxi-Tech's post 37 minutes ago
We're releasing a MAJOR update to the BananaAll SLM Super App.
If you want to use a custom architecture, previously you had to go trough reviewing the code yourself, now add an Openrouter API key and review it with GPT 6 Luna in one button. A review cost be half a cent so anyone can try it. This is one of the main features.
Now ROCm, AMD and Windows, Mac support.
Colab and Molab support.
Detailed list of features:
Get improved Windows Python detection and support paths for compatible AMD ROCm, Intel XPU, and Apple MPS setups.
Choose local training or export a self-contained Python script for Colab or Molab. Notebook runs produce a downloadable model ZIP.
Start pretraining with an existing model’s tokenizer, or train a new one from your datasets.
Try experimental 1.58-bit Ternary fake-quantized training on NVIDIA GPUs.
Watch live tokens per second. Model compilation is on by default and falls back automatically if it fails.
Build custom architectures with separate configuration and modeling files, then review the training code manually or with optional OpenRouter AI Review.
Install from source with the new coding-agent instructions.
This release also fixes inflated loss reporting for custom models.
And for those users who didn't want to try it out just because installation would be so hard, it isnt now.
Go to any coding agent (Pi, Claude Code, Codex, OpenCode, basically all work), and just paste "Install BananaAll for me. Fetch and follow https://raw.githubusercontent.com/BananaMind/BananaAll/main/agent_install.txt."
That's it.
Check it out at https://github.com/BananaMind/BananaAll/
Also on SAICR, we're currently training a new major model (NACR v2) and ACR 1.0 is in the finishing.
repliedto harshitkgupta's post about 1 hour ago
Fine-tuned Qwen 2.5 (0.5B → 3B) on real coding-agent traces, 10 controlled runs, one 16GB Mac. Compared PyTorch MPS vs. Apple MLX for local LoRA SFT — and the honest answer is "it depends on what you're optimizing for":
• PyTorch MPS: 2.2x–5.7x faster raw throughput, but hits a hard memory wall — can't load a 3B model in FP16 on 16GB.
• Apple MLX: 4-bit QLoRA fits 3B+ models with almost flat memory scaling as context grows (+109 MB going from 1k→4k tokens).
• 4-bit quantization doesn't cost you convergence — eval loss tracks closely across backends.
• The bigger surprise: most of MLX's slowdown isn't the 4-bit dequant tax. Two of the 10 runs went unquantized to isolate it — dequant only explains 1.07x–1.4x of the gap. A ~4.1–4.6x framework-level gap remains either way.
All 10 LoRA adapters + Trackio logs are public so the numbers are checkable, not just claimed.
Full writeup: https://huggingface.co/blog/harshitkgupta/fine-tuning-coding-agents-on-mac-pytorch-mps-mlx
Organizations
