AI & ML interests
None defined yet.
Recent Activity
How small can a language model get before it stops being useful?
That is the whole question. We build models from scratch to find out. Freeze the parts that do not need to learn. Cut the weights down to three values. Move memory onto the disk. Then see what is left. We publish all of it, the wins and the losses, with the code, data and checkpoints, so anyone can run it and check it.
What we have found so far
Models and data
SHADOW Assistant
One small model that reads, sees and listens. A 135M ternary body with a 4B-row memory table, trained from scratch on text, pictures, video and speech, with Australia a quarter of everything it reads. Built to run offline on a normal laptop CPU.
SHADOW 250M Instruct
Built from scratch, with a 100M-token memory on disk.
SHADOW 50M Instruct
Three-valued weights, arithmetic circuits inside the model, memory on disk.
Embeddings Are Overpriced
The three models from the paper: a frozen vocabulary is enough.
Sai Kiran Bathula, New South Wales, Australia · saikiranbathula@qlni.io
Releases are for non-commercial research and education unless a repository says otherwise.
