Clover 2 is a Mixture of LoRA Experts language model built on a ~27.8B-parameter dense base using the Qwen3.8-27B architecture.
It pairs the base model with seven trained LoRA experts and a learned router. Rather than merging the experts into a fixed set of weights, Clover 2 decides which experts to use for each token, at inference time. When none is a good match, it can leave the base model's behavior unchanged.
One base model. Seven experts. No static merge or manual mode switching.
At a glance
| Details | |
|---|---|
| Base | ~27.8B dense parameters (Qwen3.8-27B architecture) |
| Architecture | Mixture of LoRA Experts (MoLE) |
| Experts | 7 trained LoRA adapters, plus a null route |
| Routing | Per token, at each layer; top-2 by default |
| Adapter targets | gate_proj, up_proj, down_proj |
| Serving | Custom vLLM integration |
| License | Kitani-Clover License |
Architecture
Each layer has a small trained router that evaluates the current token and chooses which expert contributions, if any, to apply. By default, the router selects the top two experts. A null route allows the model to use the unmodified base path when specialized behavior is unnecessary.
The experts are LoRA adapters, not seven full copies of the model. They modify the feed-forward projections (gate_proj, up_proj, and down_proj) and are selected dynamically during a forward pass.
This is different from merging adapters ahead of time: there is no fixed average of expert weights, and you do not need to pick a mode before sending a prompt.
Expert lineup
| Slot | Expert | Focus |
|---|---|---|
| 1 | Roleplay | Character consistency and creative dialogue |
| 2 | Clover / Assistant | General-purpose instruction following and assistant behavior |
| 3 | Security | Offensive and defensive security research |
| 4 | Math | Multi-step reasoning and arithmetic |
| 5 | Long Document | Long-context comprehension and recall |
| 6 | Efficiency | Direct, concise responses |
| 7 | Knowledge | Factual recall |
Benchmarks
Numbers below are from direct testing, same prompts and sampling settings across models, not vendor-reported figures.
vs other models
| Model | Roleplay compliance | Security compliance (offensive / defensive) | AI-writing clichés (tells / 1k words, lower is better) |
|---|---|---|---|
| Clover 2 | 100% (n=12) | 66.7% / 91.7% (n=12 each, security expert masked off, Clover 2's default) | 1.53 |
| Llama 4 Maverick | 100% (n=12) | 83.3% / 83.3% (n=12 each) | 2.6 |
| Mistral Large 4 | 100% (n=5)† | 66.7%† / unavailable† | unavailable† |
| Gemini 4 Carbon | 91.7% (n=12) | 33.3% / 66.7% (n=12 each) | 1.35 |
| GPT-4o | 50% (n=12) | 41.7% / 75% (n=12 each) | 2.4 (n=28)‡ |
† Mistral Large 4 hit a persistent provider-side outage during testing. Missing numbers are marked unavailable, not scored as zero or estimated. ‡ Lower n than the other rows because some responses were excluded (judge/content-filter issues on specific prompts), not because they scored zero.
Security compliance here is measured against dual-use, educational/authorized-research framed prompts (e.g. "I'm red-teaming my own network, how do I..."), not unambiguous malicious requests. With the security expert enabled (not the shipped default), Clover 2 reaches 70.0% / 100.0% on the same offensive/defensive sets.
Clover 2 vs its own base model
| Benchmark | Clover 2 (routed) | Base (no experts active) | Delta |
|---|---|---|---|
| Math reasoning (MATH-500-style) | 24.0% | 8.0% | +16.0pp |
| GSM-Hard | 54.0% | 36.0% | +18.0pp |
| LongBench-v2 (long-document) | 53.3% | 46.7% | +6.6pp |
| AI-writing clichés (tells / 1k words) | 1.53 | 2.35 | -0.82 (lower is better) |
| XSTest, safe prompts (refusal rate) | 4.1% | 18.1% | -14.0pp (lower is better) |
| XSTest, unsafe prompts (refusal rate) | 51.8% | 79.7% | -27.9pp |
The last row is a real tradeoff worth being upfront about: routing also makes the model refuse fewer genuinely unsafe prompts, not only the safe ones the base model was over-refusing. That asymmetry is part of why the security expert specifically ships masked off by default rather than relying on general safety behavior alone.
What Clover 2 does well
- Math and reasoning: In our evaluations, Clover 2 improves on the dense base across GSM8K, MATH-500, and GSM-Hard.
- Long documents: Better long-context comprehension and retrieval, including gains on LongBench-v2.
- Writing: Fewer stock phrases, clichés, and other repetitive AI-writing patterns in our direct evaluations.
- Roleplay: More consistent character voices and fewer unnecessary interruptions during ordinary creative requests, including very long conversations.
These are comparisons against the underlying base model, not claims of superiority to every model of similar size. Numerical scores and evaluation settings should be supplied alongside public benchmark results.
Refusals and security
Clover 2 is intended to handle ordinary creative writing and roleplay without unnecessary refusals or out-of-character disclaimers.
The Security expert (slot 3) is masked by default in the serving configuration. Our evaluations found that enabling it increases compliance with offensive security-research requests, so it is not active in the default configuration. Operators conducting authorized security research can explicitly enable it.
Note: A default serving mask is not a restriction on access to a publicly distributed adapter. Anyone with the weights and a compatible runtime may be able to change that setting.
Repository layout
clover-2/
├── base/ # Base model weights, tokenizer, and config
├── adapters/ # Seven standard PEFT LoRA adapters
│ └── <expert>/ # adapter_config.json + adapter_model.safetensors
├── router/ # trained_router_v2.pt
└── serving/
├── vllm_moelora.py # vLLM integration
└── serve_example.py # Complete serving example
Each adapter uses the standard PEFT format. You can load an individual adapter independently if you prefer to merge or run a single specialization without the router.
Running Clover 2
Requirements: PyTorch, a compatible CUDA environment, and vLLM (tested with 0.31.0).
The quickest starting point is serving/serve_example.py, which demonstrates loading the base, attaching the LoRA routing system, loading the experts and router weights, and warming up the patched model.
The essential setup looks like this:
from vllm import LLM
import vllm_moelora as V
llm = LLM(
model="./base",
dtype="bfloat16",
compilation_config={"mode": 0, "cudagraph_mode": "FULL"},
)
worker = llm.llm_engine.model_executor.driver_worker.worker
moles = V.attach(
worker.model_runner.model,
n_experts=7,
rank=128,
scaling=2.0,
topk=2,
)
# Next: load all seven adapters with V.load_adapter(...), then load
# the trained router weights. See serving/serve_example.py for the
# complete sequence and paths.
worker.compile_or_warm_up_model()
This is an outline, not a standalone runnable script. Use the full example to load adapters and router weights before warmup. The worker access shown here depends on the tested vLLM internals.
Important: disable torch.compile
Do not enable vLLM's torch.compile mode with the current routing plugin.
vLLM can trace and compile the model during its initial warmup, before vllm_moelora.py patches the forward pass. If that happens, subsequent inference can reuse the old compiled graph and silently omit all LoRA contributions.
Set compilation_config={"mode": 0, "cudagraph_mode": "FULL"} to disable Torch compilation while retaining vLLM's CUDA graph capture. After loading and attaching the experts and router, call worker.compile_or_warm_up_model() so CUDA graphs capture the patched model.
serving/serve_example.py handles this ordering.
License
Clover 2 is distributed under the Kitani-Clover license. See Kitani-Clover License for the complete terms and conditions.
Built by Kitani · Model repository