MiniGrimm (48M SFT)

MiniGrimm is a tiny 48.29M-parameter (42.00M non-embedding) decoder-only Transformer trained from scratch and fine-tuned with Supervised Fine-Tuning (SFT).

This is a small-scale experimental model built to test a deep-and-thin Llama-style architecture trained with the Muon optimizer. Because it is under 50M parameters and has a 512-token context window, keep your expectations realistic—it is not a general-purpose assistant and will hallucinate facts or stumble on complex logic. That said, it can hold short casual chats and do basic step-by-step arithmetic surprisingly well for its size.


Benchmarks

Evaluated after SFT on ArithMark 3.0 (arithmetic reasoning):

Benchmark Metric Score
ArithMark 3.0 Raw Accuracy 38.1%
ArithMark 3.0 Normalized Accuracy 38.2%

(Not bad for a 42M non-embedding model, though it will still make plenty of mistakes on multi-step word problems or larger numbers.)


Other Benchmarks:

Benchmark Accuracy (Raw) Accuracy (Norm) Notes
SciQ 70.7% (±1.44) 63.0% (±1.53) Science QA
PIQA 61.7% (±1.13) 62.0% (±1.13) Physical commonsense
Winogrande 52.4% (±1.40) — Pronoun resolution
BoolQ 48.6% (±0.87) — Yes/No reading comprehension
ARC-Easy 48.6% (±1.03) 43.8% (±1.02) Grade-school science
HellaSwag 29.1% (±0.45) 31.7% (±0.46) Sentence completion
OpenBookQA 19.2% (±1.76) 30.0% (±2.05) Elementary science facts
ARC-Challenge 22.3% (±1.22) 27.2% (±1.30) Harder science reasoning
LAMBADA (OpenAI) 26.9% (±0.62) — Perplexity: 83.15 (±3.71)

Architecture & Training Summary

Instead of a wide, shallow network, MiniGrimm uses a tall, narrow layout (28 layers at d_model=384) paired with modern stability tweaks:

Spec Detail
Parameters 48.29M total (~42.00M non-embedding, tied embeddings)
Layers / Hidden Dim 28 layers, d_model = 384, ffn_hidden = 960 (SwiGLU)
Attention Grouped-Query Attention (6 query heads, 2 KV heads) + QK-Norm
Positional Encoding RoPE (theta = 10000.0)
Logit Soft-Capping 30.0
Context Window 512 tokens
Tokenizer Custom 16,384-vocab Byte-Level BPE (with single-digit number splitting)
Pretraining ~12.55B tokens (FineWeb-Edu, Cosmopedia-v2, Wikipedia, CoT/math, chat) using Muon + AdamW
SFT Assistant-only loss masking across GSM8K, SmolTalk, UltraChat-200k, Tülu-3, and No Robots

Usage

Because MiniGrimm uses custom modeling code (for QK-Norm and logit soft-capping), you need to pass trust_remote_code=True when loading the model.

Quickstart with transformers

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

repo_id = "Bayhakl1/MiniGrimm"
device = "cuda" if torch.cuda.is_available() else "cpu"

tokenizer = AutoTokenizer.from_pretrained(repo_id)
model = AutoModelForCausalLM.from_pretrained(
    repo_id,
    trust_remote_code=True,
).to(device)
model.eval()

# Format the prompt using MiniGrimm's special chat tokens
prompt = (
    "<|system|>You are a helpful assistant.<|end|>"
    "<|user|>If I have 3 apples and buy 5 more, then eat 2, how many apples do I have?<|end|>"
    "<|assistant|>"
)

inputs = tokenizer(prompt, return_tensors="pt").to(device)

# Stop when the model emits either <|end|> or <|endoftext|>
stop_token_ids = [
    tokenizer.convert_tokens_to_ids("<|end|>"),
    tokenizer.eos_token_id,
]

with torch.no_grad():
    output_ids = model.generate(
        **inputs,
        max_new_tokens=150,
        do_sample=True,
        temperature=0.6,
        top_p=0.9,
        eos_token_id=stop_token_ids,
        pad_token_id=tokenizer.pad_token_id,
    )

# Slice off the prompt tokens so only the assistant's reply is printed
reply_ids = output_ids[0][inputs["input_ids"].shape[1]:]
print(tokenizer.decode(reply_ids, skip_special_tokens=True))

Important Usage Notes

  1. Chat Template Format: Always wrap turns using the exact single-token delimiters <|system|>, <|user|>, <|assistant|>, and <|end|>:
    <|system|>Optional system prompt.<|end|><|user|>First message<|end|><|assistant|>First reply<|end|><|user|>Follow-up<|end|><|assistant|>
    
    (The <|system|>...<|end|> block is optional—you can start directly with <|user|>.)
  2. 512-Token Context Limit: The model was trained with a maximum sequence length of 512 tokens (prompt + generated tokens combined). Keep prompts short and max_new_tokens around 100–250 so the total length stays within 512 tokens.
  3. Greedy vs. Sampling: For math questions, do_sample=False (greedy decoding) usually gives cleaner step-by-step arithmetic. For casual chat, do_sample=True with temperature=0.6–0.7 helps avoid repetitive loops.

Limitations

  • Small Model Scale (~48M): Limited factual recall—it will confidently make up dates, names, or trivia.
  • Short Memory: Anything past 512 tokens is outside its training window.
  • English Only: Trained almost exclusively on English text and basic math.
Downloads last month
225
Safetensors
Model size
48.3M params
Tensor type
F32
·
Inference Providers NEW
This model isn't deployed by any Inference Provider. 🙋 Ask for provider support

Datasets used to train Bayhakl1/MiniGrimm