Indic-Delivery-Rider-SLM (40M parameters)

A small, from-scratch, Llama-style language model for offline assistance of delivery partners and drivers (e.g., Swiggy, Zomato, Porter) in India.

Languages: English, Hindi, Bengali, Marathi, Tamil.

Experimental research model. Trained for ~28 minutes on one T4 GPU on a small dataset. Non-English domain data is machine-translated and has not been reviewed by native speakers. Not for payments, safety or legal advice.

Intended Use Case

• Offline edge execution in areas with poor connectivity.
• Quick responses for address changes, payment status checks, COD handling, and order issues.

Architecture

Model Architecture

• Architecture: Decoder-only Transformer
• Total Parameters: 39.7 million
• Transformer Layers: 10
• Hidden Dimension: 512
• Attention Heads: 8
• Key-Value Heads: 4 (Grouped-Query Attention, GQA)
• Feed-Forward Network: SwiGLU-activated MLP with an intermediate dimension of 1,408
• Normalization: Root Mean Square Layer Normalization (RMSNorm)
• Positional Encoding: Rotary Position Embeddings (RoPE)
• Embedding Configuration: Tied input and output embeddings
• Maximum Context Length: 384 tokens
• Tokenizer: SentencePiece BPE with a 20,000-token vocabulary and byte fallback
• Hugging Face Compatibility: Exports as LlamaForCausalLM, with output logits verified against the original implementation.

Tokenizer fertility on held-out text:

• English: 1.28 tokens/word (GPT-2 BPE: 1.30)
• Hindi: 1.26 tokens/word (GPT-2 BPE: 7.76)
• Bengali: 1.45 tokens/word (GPT-2 BPE: 12.79)
• Marathi: 1.57 tokens/word (GPT-2BPE: 10.50)
• Tamil: 1.81 tokens/word (GPT-2 BPE: 23.44)

Prompt format

<s><|user|>{question}<|assistant|><|lang|> then the model writes the answer and </s>.

<|lang|> is one of <|en|>, <|hi|>, <|bn|>, <|mr|>, <|ta|>; prefill it to force the reply language, or omit it to let the model choose.

Usage

import torch, sentencepiece as spm
from huggingface_hub import snapshot_download
from transformers import AutoModelForCausalLM

path = snapshot_download("learner-subodh/Indic-Delivery-Rider-SLM")
sp = spm.SentencePieceProcessor(model_file=f"{path}/tokenizer.model")
model = AutoModelForCausalLM.from_pretrained(path).eval()

q = "Customer wants to change the delivery address after pickup. What should I tell them?"
ids = [sp.bos_id(), sp.piece_to_id("<|user|>")] + sp.encode(q) + [sp.piece_to_id("<|assistant|>"), sp.piece_to_id("<|en|>")]
out = model.generate(torch.tensor([ids]), max_new_tokens=160, do_sample=True, temperature=0.6, top_k=40, top_p=0.9,
                     repetition_penalty=1.15, eos_token_id=sp.eos_id(), pad_token_id=sp.pad_id())
print(sp.decode(out[0][len(ids):].tolist()))

Training data

• Bitext customer-support dataset (English, bitext/Bitext-customer-support-llm-chatbot-training-dataset)
• Sentence-level machine translation of a stratified subset into the other languages with NLLB-200 (600M distilled), filtered for placeholders, script, repetition and length
• Native-script instruction data from the Aya dataset (CohereForAI/aya_dataset, Apache-2.0)

Evaluation (held-out)

Language Modelling

Bits per byte (BPB) measures the model's answer-text prediction performance, normalised by text bytes (lower is better). Perplexity (PPL) is reported as a supplementary metric.

• domain: Support-style data.
• general: Aya dataset.
• resp_tokens: Number of response tokens evaluated.

Language	Data Split	Bits per Byte	Perplexity (PPL)	Response Tokens
Bengali (bn)	Domain	0.204	6.614	10,821
Bengali (bn)	General	0.825	193.146	5,206
English (en)	Domain	0.258	2.297	52,493
Hindi (hi)	Domain	0.199	4.506	12,750
Hindi (hi)	General	0.694	51.689	3,170
Marathi (mr)	Domain	0.217	7.363	11,822
Marathi (mr)	General	0.668	36.486	4,682
Tamil (ta)	Domain	0.177	7.704	12,050
Tamil (ta)	General	0.731	119.382	9,661

Multilingual Generation

Generation checks on hand-written rider prompts (3 per language; tiny sample, indicative only).

• lang_fidelity: reply is in the expected script (and Hindi/Marathi marker words).
• auto_lang_id: model picks the right language tag unprompted.
• distinct2: 1 - repetition (higher indicates greater lexical diversity).
• stop_rate: fraction of generations terminating with the expected stop token.
• avg_words: average generated words per response.

Language	Lang Fidelity	Auto Lang ID	Distinct-2	Stop Rate	Avg. Words
Bengali (bn)	1.00	1.00	0.95	0.67	80.00
English (en)	1.00	1.00	0.98	0.67	75.00
Hindi (hi)	1.00	1.00	0.94	1.00	91.67
Marathi (mr)	0.67	1.00	0.65	0.67	57.67
Tamil (ta)	1.00	1.00	0.99	0.67	89.67

Evaluation Scope

• Language modelling: Evaluates next-token prediction on held-out domain and general-language text; does not establish factual correctness.
• Generation: Preliminary assessment of language consistency and generation behaviour using 15 manually written prompts.
• Tokenizer: Fertility comparison measures tokenisation efficiency against GPT-2 BPE, not language understanding.
• Task performance: Intent classification Macro-F1, entity extraction F1 and task completion rate have not been independently benchmarked.
• Deployment: CPU inference latency, peak RAM, throughput and quantised inference performance have not yet been benchmarked.
• Human evaluation: No comprehensive native-speaker or human assessment has been conducted.

Limitations

• Small capacity, small data: it can state wrong policies fluently, blur Hindi and Marathi (same script), and repeat itself.
• Machine-translated training text carries translation errors.
• General-language performance is limited, as reflected in held-out perplexity.
• Generation checks use only three prompts per language and are not statistically representative.
• Route anything involving money, injuries or disputes to a human or a fixed template.

Licensing note

• Mixed provenance: Bitext (CDLA-Sharing-1.0), Aya (Apache-2.0), and translations produced by NLLB-200 (model weights CC-BY-NC-4.0).
• Check these terms with your own counsel before any commercial use.
• Weights are released for education, research and evaluation.