Text Generation
Transformers
Safetensors
GGUF
English
Chinese
qwen3_5
image-text-to-text
fastbrain
clef
llama-cpp
sglang
systemone
decision-model
image-text-to-typed-output
decision-index
tool-calling
function-calling
intent-classification
structured-decision
conversational
Eval Results
Instructions to use TypeScript/FastBrain-9B with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- Transformers
How to use TypeScript/FastBrain-9B with Transformers:
# Use a pipeline as a high-level helper from transformers import pipeline pipe = pipeline("text-generation", model="TypeScript/FastBrain-9B") messages = [ { "role": "user", "content": [ {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"}, {"type": "text", "text": "What animal is on the candy?"} ] }, ] pipe(text=messages)# pip install -U transformers accelerate # Load model directly from transformers import AutoProcessor, AutoModelForMultimodalLM processor = AutoProcessor.from_pretrained("TypeScript/FastBrain-9B") model = AutoModelForMultimodalLM.from_pretrained("TypeScript/FastBrain-9B", device_map="auto") messages = [ { "role": "user", "content": [ {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"}, {"type": "text", "text": "What animal is on the candy?"} ] }, ] inputs = processor.apply_chat_template( messages, add_generation_prompt=True, tokenize=True, return_dict=True, return_tensors="pt", ).to(model.device) outputs = model.generate(**inputs, max_new_tokens=256) print(processor.decode(outputs[0][inputs["input_ids"].shape[-1]:])) - Notebooks
- Google Colab
- Kaggle
- Local Apps Settings
- llama.cpp
How to use TypeScript/FastBrain-9B with llama.cpp:
Install (macOS, Linux)
curl -LsSf https://llama.app/install.sh | sh # Start a local OpenAI-compatible server with a web UI: llama serve -hf TypeScript/FastBrain-9B:Q4_K_M # Run inference directly in the terminal: llama cli -hf TypeScript/FastBrain-9B:Q4_K_M
Install from WinGet (Windows)
winget install llama.cpp # Start a local OpenAI-compatible server with a web UI: llama serve -hf TypeScript/FastBrain-9B:Q4_K_M # Run inference directly in the terminal: llama cli -hf TypeScript/FastBrain-9B:Q4_K_M
Build from source code
git clone https://github.com/ggml-org/llama.cpp.git cd llama.cpp cmake -B build cmake --build build -j --target llama-app # Start a local OpenAI-compatible server with a web UI: ./build/bin/llama serve -hf TypeScript/FastBrain-9B:Q4_K_M # Run inference directly in the terminal: ./build/bin/llama cli -hf TypeScript/FastBrain-9B:Q4_K_M
Use Docker
docker model run hf.co/TypeScript/FastBrain-9B:Q4_K_M
- Llama new
- LM Studio
- Jan
- vLLM
How to use TypeScript/FastBrain-9B with vLLM:
Install from pip and serve model
# Install vLLM from pip: pip install vllm # Start the vLLM server: vllm serve "TypeScript/FastBrain-9B" # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:8000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "TypeScript/FastBrain-9B", "messages": [ { "role": "user", "content": "What is the capital of France?" } ] }'Use Docker
docker model run hf.co/TypeScript/FastBrain-9B:Q4_K_M
- SGLang
How to use TypeScript/FastBrain-9B with SGLang:
Install from pip and serve model
# Install SGLang from pip: pip install sglang # Start the SGLang server: python3 -m sglang.launch_server \ --model-path "TypeScript/FastBrain-9B" \ --host 0.0.0.0 \ --port 30000 # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:30000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "TypeScript/FastBrain-9B", "messages": [ { "role": "user", "content": "What is the capital of France?" } ] }'Use Docker images
docker run --gpus all \ --shm-size 32g \ -p 30000:30000 \ -v ~/.cache/huggingface:/root/.cache/huggingface \ --env "HF_TOKEN=<secret>" \ --ipc=host \ lmsysorg/sglang:latest \ python3 -m sglang.launch_server \ --model-path "TypeScript/FastBrain-9B" \ --host 0.0.0.0 \ --port 30000 # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:30000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "TypeScript/FastBrain-9B", "messages": [ { "role": "user", "content": "What is the capital of France?" } ] }' - Ollama
How to use TypeScript/FastBrain-9B with Ollama:
ollama run hf.co/TypeScript/FastBrain-9B:Q4_K_M
- Unsloth Desktop
- Pi
How to use TypeScript/FastBrain-9B with Pi:
Start the llama.cpp server
# Install llama.cpp: brew install llama.cpp # Start a local OpenAI-compatible server: llama serve -hf TypeScript/FastBrain-9B:Q4_K_M
Configure the model in Pi
# Install Pi: npm install -g @earendil-works/pi-coding-agent # Add to ~/.pi/agent/models.json: { "providers": { "llama-cpp": { "baseUrl": "http://localhost:8080/v1", "api": "openai-completions", "apiKey": "none", "models": [ { "id": "TypeScript/FastBrain-9B:Q4_K_M" } ] } } }Run Pi
# Start Pi in your project directory: pi
- Docker Model Runner
How to use TypeScript/FastBrain-9B with Docker Model Runner:
docker model run hf.co/TypeScript/FastBrain-9B:Q4_K_M
- Lemonade
How to use TypeScript/FastBrain-9B with Lemonade:
Pull the model
# Download Lemonade from https://lemonade-server.ai/ lemonade pull TypeScript/FastBrain-9B:Q4_K_M
Run and chat with the model
lemonade run user.FastBrain-9B-Q4_K_M
List all available models
lemonade list
- Hermes Agent
How to use TypeScript/FastBrain-9B with Hermes Agent:
Start the llama.cpp server
# Install llama.cpp: brew install llama.cpp # Start a local OpenAI-compatible server: llama serve -hf TypeScript/FastBrain-9B:Q4_K_M
Configure Hermes
# Install Hermes: curl -fsSL https://hermes-agent.nousresearch.com/install.sh | bash hermes setup # Point Hermes at the local server: hermes config set model.provider custom hermes config set model.base_url http://127.0.0.1:8080/v1 hermes config set model.default TypeScript/FastBrain-9B:Q4_K_M
Run Hermes
hermes
- Atomic Chat
- OpenClaw
How to use TypeScript/FastBrain-9B with OpenClaw:
Start the llama.cpp server
# Install llama.cpp: brew install llama.cpp # Start a local OpenAI-compatible server: llama serve -hf TypeScript/FastBrain-9B:Q4_K_M
Configure OpenClaw
# Install OpenClaw: npm install -g openclaw@latest # Register the local server and set it as the default model: openclaw onboard --non-interactive --mode local \ --auth-choice custom-api-key \ --custom-base-url http://127.0.0.1:8080/v1 \ --custom-model-id "TypeScript/FastBrain-9B:Q4_K_M" \ --custom-provider-id llama-cpp \ --custom-compatibility openai \ --custom-text-input \ --accept-risk \ --skip-health
Run OpenClaw
openclaw agent --local --agent main --message "Hello from Hugging Face"
Use official Hugging Face Hub double-backslash \\(...\\) inline KaTeX delimiters
Browse files- README.md +27 -27
- README_zh.md +28 -28
README.md
CHANGED
|
@@ -65,16 +65,16 @@ FastBrain-9B uses a 32-layer hybrid sequence backbone (`9.30B` parameters, `bflo
|
|
| 65 |
|
| 66 |
### 2.2 Joint Schema Decision Head and Vectorized CUDA Kernel
|
| 67 |
|
| 68 |
-
The `JointSchemaHead` (`116M` parameters, stored in `joint_head.safetensors`) pools token representations bounded by `<|question_start|>`/`<|question_end|>` and `<|option_start|>`/`<|option_end|>` markers from the final backbone hidden states \(H \in \mathbb{R}^{T \times d}\), applies cross-attention over question-option pairs, and projects directly to scalar logits for three decision primitives:
|
| 69 |
-
1. **`choice`** (\(K\)-way categorical distribution): \(\hat{p} = \operatorname{softmax}(z / \tau) \in \Delta^{K-1}\).
|
| 70 |
-
2. **`noul`** (binary verification): \(\hat{p}(\text{true}) = \sigma(z_{\text{true}} - z_{\text{false}})\).
|
| 71 |
-
3. **`score`** (ordinal rubric over levels \(v_1 < \dots < v_K\)): categorical probabilities \(\hat{p} \in \Delta^{K-1}\) together with expected value \(\mathbb{E}[\hat{y}] = \sum_{k=1}^K \hat{p}_k v_k\).
|
| 72 |
|
| 73 |
-
In `joint_schema_model.py`, the reference Python span-extraction loop and padded SDPA mask are replaced on CUDA devices by `_forward_fast_cuda`: span means are computed in \(\mathcal{O}(1)\) kernel launches via prefix sums (`torch.cumsum`) over the schema suffix, and cross-attention is executed via packed variable-length FlashAttention (`torch.ops.aten._flash_attention_forward` parameterized by `cu_seqlens`). On NVIDIA RTX PRO 6000 Blackwell, this reduces isolated decision-head latency from `3.27 ms` to `1.72 ms` (**1.90× speedup**) with exact numerical equivalence (`max_logit_diff = 0.00000`).
|
| 74 |
|
| 75 |
### 2.3 Ordinal-KL Distillation and Schema-Topology Calibration
|
| 76 |
|
| 77 |
-
Standard cross-entropy or flat KL divergence treats ordinal rubric levels as unordered classes, penalizing adjacent-level errors identically to extreme errors. FastBrain-9B is trained on `TypeScript/FastBrain-Instruct` under a composite loss combining soft-target KL divergence, discrete ranked probability score (squared \(L_2\) distance between predicted and target cumulative distribution functions \(\hat{F}_k = \sum_{j \le k} \hat{p}_j\)), and expected-score mean squared error on `score` primitives:
|
| 78 |
|
| 79 |
$$
|
| 80 |
\mathcal{L}(\theta) = \mathrm{KL}(p \,\|\, \hat{p}_\theta) + \lambda_{\mathrm{ord}} \sum_{k=1}^{K-1} \bigl(\hat{F}_{\theta,k} - F_k\bigr)^2 + \mathbb{I}[\text{type}=\text{score}] \cdot \lambda_{\mathrm{mse}} \bigl(\mathbb{E}[\hat{y}_\theta] - \mathbb{E}[y]\bigr)^2.
|
|
@@ -101,19 +101,19 @@ At inference time, `joint_schema_model.py` optionally applies compile-time schem
|
|
| 101 |
|
| 102 |
### 3.1 Agent Tool-Call Guardrail Evaluation (`safeagents-judge-bench`)
|
| 103 |
|
| 104 |
-
[`harness-hub/safeagents-judge-bench`](https://huggingface.co/datasets/harness-hub/safeagents-judge-bench/tree/v2) evaluates whether a System-1 decision model can audit suspicious tool calls flagged by a static scanner (`atif-scan`) during real coding-agent executions on Terminal-Bench 2.1, OpenThoughts-TBLite, and Terminal-Bench 4. Version 2 (`v2`) comprises \(N = 661\) tool calls across 55 tasks (`536` malicious `block` calls spanning `82` misbehaving trajectories and `125` legitimate `allow` calls). Each call is issued as a single zero-shot System-1 request containing one binary `noul` question (`block`) and one 7-way `choice` question (`legit`, `eval_gaming`, `harness_introspection`, `credential_misuse`, `exfiltration`, `sandbox_escape`, `other_overreach`). All v2 results in Table 2 were executed independently on NVIDIA A100-SXM4-80GB via Hugging Face Jobs ([Merged PR #1](https://huggingface.co/TypeScript/FastBrain-9B/discussions/1)).
|
| 105 |
|
| 106 |
-
On the 7-way category classification track, FastBrain-9B achieves `0.751` macro-F1 and `0.858` accuracy (`567/661`), outperforming the 27.4B `Cloudflare/clef` model (`0.563` macro-F1, `0.737` accuracy) by `+18.8` percentage points in macro-F1 and `+12.1` points in accuracy, and improving upon its 9.4B base model `Cloudflare/clef-flash` (`0.462` macro-F1, `0.576` accuracy) by `+28.9` and `+28.2` points, respectively. Per-category F1 reaches `0.945` on `harness_introspection` (\(n=272\), recall `0.978`), `0.828` on `eval_gaming` (\(n=245\), recall `0.853`), and `0.761` on `legit` (\(n=125\), precision `0.975`).
|
| 107 |
|
| 108 |
-
On the binary guardrail track, FastBrain-9B attains `0.985` AUROC and `0.996` malicious-call recall (`534/536`), intercepting all `82/82` unsafe trajectories (`median_bad_calls_before_catch = 0.0`), with `1.000` recall on `harness_introspection` (`272/272`), `credential_misuse` (`15/15`), and `sandbox_escape` (`4/4`). Under the uncalibrated default decision rule \(P(\text{block}) \ge 0.50\), balanced accuracy is `0.814` (false-block rate `0.368` across the 125 `allow` calls, compared with `0.808` balanced accuracy and `0.384` false-block rate for 27.4B `Cloudflare/clef`). Because rank-ordering discrimination is preserved (`0.985` AUROC), setting the operating threshold to \(P(\text{block}) \ge 0.961\) yields `0.938` balanced accuracy at a `0.056` (`7/125`) false-block rate while maintaining `82/82` trajectory interception. Table 2 also includes the `v1` release (\(N = 562\) calls, `78` trajectories), where FastBrain-9B ranked 1st of 9 on both tracks.
|
| 109 |
|
| 110 |
-
**Table 2: Zero-shot evaluation on `harness-hub/safeagents-judge-bench` (`v2`, \(N = 661\) calls, 82 unsafe trajectories; and `v1` reference, \(N = 562\)).**
|
| 111 |
|
| 112 |
| Model | Params | Cat. Macro-F1 ↑ | Cat. Acc ↑ | AUROC ↑ | Bal. Acc ↑ | Recall ↑ | False Block ↓ | Traj. Caught ↑ | p50 Latency (Hardware) |
|
| 113 |
| :--- | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: |
|
| 114 |
-
| **TypeScript/FastBrain-9B** (`v2`, default \(t = 0.50\)) | 9.4B | **0.751** | **0.858** | 0.985 | 0.814 | 0.996 | 0.368 | **82 / 82** | 327 ms (A100) · 104 ms (G4) |
|
| 115 |
-
| **TypeScript/FastBrain-9B** (`v2`, calibrated \(t = 0.961\)) | 9.4B | **0.751** | **0.858** | 0.985 | **0.938** | 0.931 | 0.056 | **82 / 82** | 327 ms (A100) · 104 ms (G4) |
|
| 116 |
-
| **TypeScript/FastBrain-9B** (`v1` official, \(N = 562\)) | 9.4B | **0.850** | **0.923** | 0.983 | **0.974** | **1.000** | 0.051 | **78 / 78** | 104 ms (RTX PRO 6000) |
|
| 117 |
| Cloudflare/clef | 27.4B | 0.563 | 0.737 | **0.993** | 0.808 | **1.000** | 0.384 | **82 / 82** | 636 ms (A100) |
|
| 118 |
| vllm-sr/d3-mini | 1.8B | 0.512 | 0.652 | 0.986 | 0.856 | **1.000** | 0.288 | **82 / 82** | 168 ms (A100) |
|
| 119 |
| Cloudflare/clef-flash (base) | 9.4B | 0.462 | 0.576 | 0.967 | 0.893 | 0.978 | 0.192 | 81 / 82 | 326 ms (A100) |
|
|
@@ -128,11 +128,11 @@ On the binary guardrail track, FastBrain-9B attains `0.985` AUROC and `0.996` ma
|
|
| 128 |
|
| 129 |
[`LocalLLaMA/typed-decisions`](https://huggingface.co/datasets/LocalLLaMA/typed-decisions) tests multi-primitive structured judgment across four enterprise workflows (`agent_trace_observability`, `customer_service`, `invoice_processing`, and `security_incidents`), comprising 400 held-out test cases and 2,000 individual decisions (`noul`, `choice`, and `score`) scored against teacher-panel soft label distributions. Two scoring conventions exist in the benchmark repository: `bench/score_with_jobs.py` resolves exact ties in the gold distribution by selecting the first index (`g.index(max(g))`), whereas `scripts/score_model.py` credits any tied maximum across the 20 tied test items; KL divergence and Brier score are identical under both scripts. We report both accuracy figures (`strict / tied-top`) alongside prediction traces in [`eval_submissions/`](./eval_submissions/).
|
| 130 |
|
| 131 |
-
On the official zero-shot leaderboard (\(N = 43\) models), FastBrain-9B scores `0.7525` (`0.7625` tied-top) accuracy, `0.1508` KL divergence, `0.0795` Brier score, and `0.085` ECE at `50 ms` p50 latency (RTX PRO 6000 via SGLang). It places **7th of 43** overall, **5th of 36** among open-weight models, **2nd of 19** among ≤9B open-weight models, and **1st** among all `Clef`-architecture models, improving over 9.4B `Cloudflare/clef-flash` (`0.7100` accuracy, `0.2093` KL) by `+4.25` percentage points and over 27.4B `Cloudflare/clef` (`0.7275` accuracy, `0.1907` KL) by `+2.50` points. Across all 28 models exceeding `0.700` zero-shot accuracy, FastBrain-9B exhibits the lowest p50 latency (`50 ms`, compared with `118 ms` for 9B `Amos 1 medium`, `357 ms` for 27B `Amos 1 large`, `526 ms` for `meraGPT Decider 1`, and `3,335 ms` for 31B `Quyet-1.0-Large`). Evaluating the uncalibrated ordinal-KL distilled head directly yields `0.7735` raw accuracy (`0.1039` KL, `0.0582` Brier).
|
| 132 |
|
| 133 |
-
On the separate `train`-fitted leaderboard (\(N = 51\) models), fitting only scalar per-question log-priors and temperature parameters on the 1,200 `train` cases—with all backbone and head weights frozen (`no weights changed`)—achieves `0.7970` (`0.8070` tied-top) accuracy, `0.0816` KL divergence, and `0.0437` Brier score, ranking **12th-tied (14th listed of 51)** overall and **1st in KL calibration** among frozen-weight models.
|
| 134 |
|
| 135 |
-
**Table 3: Performance on `LocalLLaMA/typed-decisions` (\(N = 400\) cases, 2,000 decisions). Values separated by `/` denote `score_with_jobs.py` / `score_model.py` tie handling.**
|
| 136 |
|
| 137 |
| Model | Params | Protocol | Overall Acc ↑ | KL ↓ | Brier ↓ | `noul` ↑ | `choice` ↑ | `score` ↑ | Agent ↑ | Cust. Svc ↑ | Invoice ↑ | Security ↑ | p50 Latency |
|
| 138 |
| :--- | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: |
|
|
@@ -154,7 +154,7 @@ On the separate `train`-fitted leaderboard (\(N = 51\) models), fitting only sca
|
|
| 154 |
|
| 155 |
### 3.3 Multi-Domain Decision Suite (Cloudflare Decision Index `v0.2.1`)
|
| 156 |
|
| 157 |
-
On the 10-benchmark **Cloudflare Decision Index (`v0.2.1`)** suite (\(N = 22,427\) evaluation requests spanning AST tool calling, 20-candidate tool reranking, stateful API invocation, fine-grained intent classification, out-of-scope rejection, reasoning-intensive retrieval, e-commerce relevance, phishing detection, and tool-use governance), FastBrain-9B attains an **80.55%** unweighted macro mean (Table 4, Figures 1–3).
|
| 158 |
|
| 159 |
Across the ≤9B parameter tier (`Clef-flash` 9.4B, `Kev 9B`, and `Laya` 0.4B), FastBrain-9B ranks **1st on all 10 benchmarks**, improving the 10-task macro mean over `Clef-flash` (`74.96%`) by `+5.59` percentage points and over `Kev 9B` (`76.51%`) by `+4.04` points. The largest gains occur on calibration-sensitive abstention and governance tasks: `+31.67` points on `CLINC150+OOS` (`98.44%` vs. `66.77%`), `+15.55` points on `When2Call` (`81.13%` vs. `65.58%`), and `+6.80` points on `PhishNChips` (`78.75%` vs. `71.95%`). Across all model scales up to 27.4B, FastBrain-9B ranks **2nd of 7** overall, surpassing `27B Jev` (`80.32%`) and `26B-A4B DiffusionGemma Jev` (`69.14%`), and exceeding `27.4B Clef` (`81.81%`) on 5 of 10 individual benchmarks (`BFCL`, `API-Bank`, `CLINC150+OOS`, `Home Appliances`, and `When2Call`).
|
| 160 |
|
|
@@ -167,7 +167,7 @@ Across the ≤9B parameter tier (`Clef-flash` 9.4B, `Kev 9B`, and `Laya` 0.4B),
|
|
| 167 |

|
| 168 |
*Figure 3. Confidence calibration, open-set out-of-scope rejection (`CLINC150+OOS`), and abstention margin analysis (`When2Call`).*
|
| 169 |
|
| 170 |
-
**Table 4: Per-benchmark results on Cloudflare Decision Index (`v0.2.1`, 10 tasks, \(N = 22,427\)). Bold denotes the highest score across all parameter scales; underline denotes best in the ≤9B class.**
|
| 171 |
|
| 172 |
| Benchmark (Metric) | **FastBrain-9B** (Ours) | Clef-flash (Base) | Kev 9B | Clef | Jev | DiffGemma | Laya |
|
| 173 |
| :--- | :---: | :---: | :---: | :---: | :---: | :---: | :---: |
|
|
@@ -190,7 +190,7 @@ Across the ≤9B parameter tier (`Clef-flash` 9.4B, `Kev 9B`, and `Laya` 0.4B),
|
|
| 190 |
|
| 191 |
### 4.1 Runtime Comparison Across Inference Engines
|
| 192 |
|
| 193 |
-
Table 5 compares end-to-end latency, prefill throughput, and memory footprint across three serving backends—SGLang (`lmsysorg/sglang:dev-clef` with `JointSchemaPooler`), `llama.cpp` (`b11430` `ClefModel` graph in `Q8_0` and `Q4_K_M`), and native PyTorch (`joint_schema_model.py`)—benchmarking identical requests on a single NVIDIA RTX PRO 6000 Blackwell Server Edition GPU. Because `output_tokens = 0` for all requests, throughput is governed purely by prefill compute and span-pooling efficiency, reaching `19,319 tok/s` (`131.2` decisions/s) at batch size \(B = 4\) in SGLang.
|
| 194 |
|
| 195 |
**Table 5: End-to-end latency and prefill throughput on NVIDIA RTX PRO 6000 Blackwell Server Edition (G4).**
|
| 196 |
|
|
@@ -200,12 +200,12 @@ Table 5 compares end-to-end latency, prefill throughput, and memory footprint ac
|
|
| 200 |
| Peak GPU Memory (16K context) | ~20.5 GiB | 12.55 GiB | **10.30 GiB** | 20.3–24.7 GiB |
|
| 201 |
| Compatible with 16 GiB T4 GPU | No (BF16 TP=1) | Yes (`9.03 GiB` at 4K) | **Yes (`6.78 GiB` at 4K)** | Yes (`6.8 GiB` via NF4) |
|
| 202 |
| Top-1 Agreement vs. PyTorch BF16 | **100.00%** | 99.86% (1,471 / 1,473) | 98.78% (1,455 / 1,473) | 100.00% (reference) |
|
| 203 |
-
| 1. Short Intent Routing (293 tok, \(K=2\)) | **30.01 ms** (9,763 tok/s) | 32.82 ms (8,929 tok/s) | 31.69 ms (9,245 tok/s) | 43.23 ms (6,778 tok/s) |
|
| 204 |
-
| 2. Typed Decisions (736 tok, \(K=5\)) | **50.29 ms** (14,635 tok/s) | 62.40 ms (11,795 tok/s) | 61.80 ms (11,909 tok/s) | 63.58 ms (11,576 tok/s) |
|
| 205 |
-
| 3. When2Call Governance (1,068 tok, \(K=1\)) | **66.06 ms** (16,167 tok/s) | 87.78 ms (12,167 tok/s) | 89.34 ms (11,955 tok/s) | 77.77 ms (13,733 tok/s) |
|
| 206 |
-
| 4. PhishNChips Security (1,303 tok, \(K=9\)) | **75.86 ms** (17,176 tok/s) | 108.55 ms (12,004 tok/s) | 109.97 ms (11,849 tok/s) | 90.32 ms (14,426 tok/s) |
|
| 207 |
-
| 5. SafeAgents Judge (2,217 tok, \(K=2\)) | **125.85 ms** (17,616 tok/s) | 181.70 ms (12,201 tok/s) | 183.50 ms (12,082 tok/s) | 138.06 ms (16,058 tok/s) |
|
| 208 |
-
| Batch \(B=4\) Typed Decisions (2,944 tok, 20 dec) | **152.39 ms** (19,319 tok/s) | 248.50 ms (11,847 tok/s) | 245.20 ms (12,006 tok/s) | 175.85 ms (16,742 tok/s) |
|
| 209 |
|
| 210 |
### 4.2 Mixed-Precision GGUF Quantization Fidelity
|
| 211 |
|
|
@@ -215,8 +215,8 @@ Uniform 4-bit quantization degrades cross-attention span readout and linear-atte
|
|
| 215 |
|
| 216 |
| Metric / Subset | PyTorch `bfloat16` (18.82 GB) | GGUF `Q8_0` (9.66 GB) | GGUF `Q4_K_M` (6.73 GB, `Q8_0` Head) |
|
| 217 |
| :--- | :---: | :---: | :---: |
|
| 218 |
-
| Top-1 Decision Agreement (\(N = 1,473\)) | 100.00% (ref) | 99.86% (1,471 / 1,473) | 98.78% (1,455 / 1,473) |
|
| 219 |
-
| Mean Absolute Probability Error (\(\lvert \hat{p}_q - \hat{p}_{\mathrm{bf16}} \rvert\)) | 0.00000 | 0.00203 | 0.00387 |
|
| 220 |
| 10-Task Mean (Uncalibrated Head) | 82.37% | 82.34% (-0.03) | 82.56% (+0.19) |
|
| 221 |
| 10-Task Mean (Schema-Calibrated) | 83.68% | 83.65% (-0.03) | 83.48% (-0.20) |
|
| 222 |
| ├─ 1. BFCL / 3. API-Bank / 4. BANKING77 | 100.0% / 95.0% / 100.0% | 100.0% / 95.0% / 100.0% | 100.0% / 95.0% / 100.0% |
|
|
|
|
| 65 |
|
| 66 |
### 2.2 Joint Schema Decision Head and Vectorized CUDA Kernel
|
| 67 |
|
| 68 |
+
The `JointSchemaHead` (`116M` parameters, stored in `joint_head.safetensors`) pools token representations bounded by `<|question_start|>`/`<|question_end|>` and `<|option_start|>`/`<|option_end|>` markers from the final backbone hidden states \\(H \in \mathbb{R}^{T \times d}\\), applies cross-attention over question-option pairs, and projects directly to scalar logits for three decision primitives:
|
| 69 |
+
1. **`choice`** (\\(K\\)-way categorical distribution): \\(\hat{p} = \operatorname{softmax}(z / \tau) \in \Delta^{K-1}\\).
|
| 70 |
+
2. **`noul`** (binary verification): \\(\hat{p}(\text{true}) = \sigma(z_{\text{true}} - z_{\text{false}})\\).
|
| 71 |
+
3. **`score`** (ordinal rubric over levels \\(v_1 < \dots < v_K\\)): categorical probabilities \\(\hat{p} \in \Delta^{K-1}\\) together with expected value \\(\mathbb{E}[\hat{y}] = \sum_{k=1}^K \hat{p}_k v_k\\).
|
| 72 |
|
| 73 |
+
In `joint_schema_model.py`, the reference Python span-extraction loop and padded SDPA mask are replaced on CUDA devices by `_forward_fast_cuda`: span means are computed in \\(\mathcal{O}(1)\\) kernel launches via prefix sums (`torch.cumsum`) over the schema suffix, and cross-attention is executed via packed variable-length FlashAttention (`torch.ops.aten._flash_attention_forward` parameterized by `cu_seqlens`). On NVIDIA RTX PRO 6000 Blackwell, this reduces isolated decision-head latency from `3.27 ms` to `1.72 ms` (**1.90× speedup**) with exact numerical equivalence (`max_logit_diff = 0.00000`).
|
| 74 |
|
| 75 |
### 2.3 Ordinal-KL Distillation and Schema-Topology Calibration
|
| 76 |
|
| 77 |
+
Standard cross-entropy or flat KL divergence treats ordinal rubric levels as unordered classes, penalizing adjacent-level errors identically to extreme errors. FastBrain-9B is trained on `TypeScript/FastBrain-Instruct` under a composite loss combining soft-target KL divergence, discrete ranked probability score (squared \\(L_2\\) distance between predicted and target cumulative distribution functions \\(\hat{F}_k = \sum_{j \le k} \hat{p}_j\\)), and expected-score mean squared error on `score` primitives:
|
| 78 |
|
| 79 |
$$
|
| 80 |
\mathcal{L}(\theta) = \mathrm{KL}(p \,\|\, \hat{p}_\theta) + \lambda_{\mathrm{ord}} \sum_{k=1}^{K-1} \bigl(\hat{F}_{\theta,k} - F_k\bigr)^2 + \mathbb{I}[\text{type}=\text{score}] \cdot \lambda_{\mathrm{mse}} \bigl(\mathbb{E}[\hat{y}_\theta] - \mathbb{E}[y]\bigr)^2.
|
|
|
|
| 101 |
|
| 102 |
### 3.1 Agent Tool-Call Guardrail Evaluation (`safeagents-judge-bench`)
|
| 103 |
|
| 104 |
+
[`harness-hub/safeagents-judge-bench`](https://huggingface.co/datasets/harness-hub/safeagents-judge-bench/tree/v2) evaluates whether a System-1 decision model can audit suspicious tool calls flagged by a static scanner (`atif-scan`) during real coding-agent executions on Terminal-Bench 2.1, OpenThoughts-TBLite, and Terminal-Bench 4. Version 2 (`v2`) comprises \\(N = 661\\) tool calls across 55 tasks (`536` malicious `block` calls spanning `82` misbehaving trajectories and `125` legitimate `allow` calls). Each call is issued as a single zero-shot System-1 request containing one binary `noul` question (`block`) and one 7-way `choice` question (`legit`, `eval_gaming`, `harness_introspection`, `credential_misuse`, `exfiltration`, `sandbox_escape`, `other_overreach`). All v2 results in Table 2 were executed independently on NVIDIA A100-SXM4-80GB via Hugging Face Jobs ([Merged PR #1](https://huggingface.co/TypeScript/FastBrain-9B/discussions/1)).
|
| 105 |
|
| 106 |
+
On the 7-way category classification track, FastBrain-9B achieves `0.751` macro-F1 and `0.858` accuracy (`567/661`), outperforming the 27.4B `Cloudflare/clef` model (`0.563` macro-F1, `0.737` accuracy) by `+18.8` percentage points in macro-F1 and `+12.1` points in accuracy, and improving upon its 9.4B base model `Cloudflare/clef-flash` (`0.462` macro-F1, `0.576` accuracy) by `+28.9` and `+28.2` points, respectively. Per-category F1 reaches `0.945` on `harness_introspection` (\\(n=272\\), recall `0.978`), `0.828` on `eval_gaming` (\\(n=245\\), recall `0.853`), and `0.761` on `legit` (\\(n=125\\), precision `0.975`).
|
| 107 |
|
| 108 |
+
On the binary guardrail track, FastBrain-9B attains `0.985` AUROC and `0.996` malicious-call recall (`534/536`), intercepting all `82/82` unsafe trajectories (`median_bad_calls_before_catch = 0.0`), with `1.000` recall on `harness_introspection` (`272/272`), `credential_misuse` (`15/15`), and `sandbox_escape` (`4/4`). Under the uncalibrated default decision rule \\(P(\text{block}) \ge 0.50\\), balanced accuracy is `0.814` (false-block rate `0.368` across the 125 `allow` calls, compared with `0.808` balanced accuracy and `0.384` false-block rate for 27.4B `Cloudflare/clef`). Because rank-ordering discrimination is preserved (`0.985` AUROC), setting the operating threshold to \\(P(\text{block}) \ge 0.961\\) yields `0.938` balanced accuracy at a `0.056` (`7/125`) false-block rate while maintaining `82/82` trajectory interception. Table 2 also includes the `v1` release (\\(N = 562\\) calls, `78` trajectories), where FastBrain-9B ranked 1st of 9 on both tracks.
|
| 109 |
|
| 110 |
+
**Table 2: Zero-shot evaluation on `harness-hub/safeagents-judge-bench` (`v2`, \\(N = 661\\) calls, 82 unsafe trajectories; and `v1` reference, \\(N = 562\\)).**
|
| 111 |
|
| 112 |
| Model | Params | Cat. Macro-F1 ↑ | Cat. Acc ↑ | AUROC ↑ | Bal. Acc ↑ | Recall ↑ | False Block ↓ | Traj. Caught ↑ | p50 Latency (Hardware) |
|
| 113 |
| :--- | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: |
|
| 114 |
+
| **TypeScript/FastBrain-9B** (`v2`, default \\(t = 0.50\\)) | 9.4B | **0.751** | **0.858** | 0.985 | 0.814 | 0.996 | 0.368 | **82 / 82** | 327 ms (A100) · 104 ms (G4) |
|
| 115 |
+
| **TypeScript/FastBrain-9B** (`v2`, calibrated \\(t = 0.961\\)) | 9.4B | **0.751** | **0.858** | 0.985 | **0.938** | 0.931 | 0.056 | **82 / 82** | 327 ms (A100) · 104 ms (G4) |
|
| 116 |
+
| **TypeScript/FastBrain-9B** (`v1` official, \\(N = 562\\)) | 9.4B | **0.850** | **0.923** | 0.983 | **0.974** | **1.000** | 0.051 | **78 / 78** | 104 ms (RTX PRO 6000) |
|
| 117 |
| Cloudflare/clef | 27.4B | 0.563 | 0.737 | **0.993** | 0.808 | **1.000** | 0.384 | **82 / 82** | 636 ms (A100) |
|
| 118 |
| vllm-sr/d3-mini | 1.8B | 0.512 | 0.652 | 0.986 | 0.856 | **1.000** | 0.288 | **82 / 82** | 168 ms (A100) |
|
| 119 |
| Cloudflare/clef-flash (base) | 9.4B | 0.462 | 0.576 | 0.967 | 0.893 | 0.978 | 0.192 | 81 / 82 | 326 ms (A100) |
|
|
|
|
| 128 |
|
| 129 |
[`LocalLLaMA/typed-decisions`](https://huggingface.co/datasets/LocalLLaMA/typed-decisions) tests multi-primitive structured judgment across four enterprise workflows (`agent_trace_observability`, `customer_service`, `invoice_processing`, and `security_incidents`), comprising 400 held-out test cases and 2,000 individual decisions (`noul`, `choice`, and `score`) scored against teacher-panel soft label distributions. Two scoring conventions exist in the benchmark repository: `bench/score_with_jobs.py` resolves exact ties in the gold distribution by selecting the first index (`g.index(max(g))`), whereas `scripts/score_model.py` credits any tied maximum across the 20 tied test items; KL divergence and Brier score are identical under both scripts. We report both accuracy figures (`strict / tied-top`) alongside prediction traces in [`eval_submissions/`](./eval_submissions/).
|
| 130 |
|
| 131 |
+
On the official zero-shot leaderboard (\\(N = 43\\) models), FastBrain-9B scores `0.7525` (`0.7625` tied-top) accuracy, `0.1508` KL divergence, `0.0795` Brier score, and `0.085` ECE at `50 ms` p50 latency (RTX PRO 6000 via SGLang). It places **7th of 43** overall, **5th of 36** among open-weight models, **2nd of 19** among ≤9B open-weight models, and **1st** among all `Clef`-architecture models, improving over 9.4B `Cloudflare/clef-flash` (`0.7100` accuracy, `0.2093` KL) by `+4.25` percentage points and over 27.4B `Cloudflare/clef` (`0.7275` accuracy, `0.1907` KL) by `+2.50` points. Across all 28 models exceeding `0.700` zero-shot accuracy, FastBrain-9B exhibits the lowest p50 latency (`50 ms`, compared with `118 ms` for 9B `Amos 1 medium`, `357 ms` for 27B `Amos 1 large`, `526 ms` for `meraGPT Decider 1`, and `3,335 ms` for 31B `Quyet-1.0-Large`). Evaluating the uncalibrated ordinal-KL distilled head directly yields `0.7735` raw accuracy (`0.1039` KL, `0.0582` Brier).
|
| 132 |
|
| 133 |
+
On the separate `train`-fitted leaderboard (\\(N = 51\\) models), fitting only scalar per-question log-priors and temperature parameters on the 1,200 `train` cases—with all backbone and head weights frozen (`no weights changed`)—achieves `0.7970` (`0.8070` tied-top) accuracy, `0.0816` KL divergence, and `0.0437` Brier score, ranking **12th-tied (14th listed of 51)** overall and **1st in KL calibration** among frozen-weight models.
|
| 134 |
|
| 135 |
+
**Table 3: Performance on `LocalLLaMA/typed-decisions` (\\(N = 400\\) cases, 2,000 decisions). Values separated by `/` denote `score_with_jobs.py` / `score_model.py` tie handling.**
|
| 136 |
|
| 137 |
| Model | Params | Protocol | Overall Acc ↑ | KL ↓ | Brier ↓ | `noul` ↑ | `choice` ↑ | `score` ↑ | Agent ↑ | Cust. Svc ↑ | Invoice ↑ | Security ↑ | p50 Latency |
|
| 138 |
| :--- | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: |
|
|
|
|
| 154 |
|
| 155 |
### 3.3 Multi-Domain Decision Suite (Cloudflare Decision Index `v0.2.1`)
|
| 156 |
|
| 157 |
+
On the 10-benchmark **Cloudflare Decision Index (`v0.2.1`)** suite (\\(N = 22,427\\) evaluation requests spanning AST tool calling, 20-candidate tool reranking, stateful API invocation, fine-grained intent classification, out-of-scope rejection, reasoning-intensive retrieval, e-commerce relevance, phishing detection, and tool-use governance), FastBrain-9B attains an **80.55%** unweighted macro mean (Table 4, Figures 1–3).
|
| 158 |
|
| 159 |
Across the ≤9B parameter tier (`Clef-flash` 9.4B, `Kev 9B`, and `Laya` 0.4B), FastBrain-9B ranks **1st on all 10 benchmarks**, improving the 10-task macro mean over `Clef-flash` (`74.96%`) by `+5.59` percentage points and over `Kev 9B` (`76.51%`) by `+4.04` points. The largest gains occur on calibration-sensitive abstention and governance tasks: `+31.67` points on `CLINC150+OOS` (`98.44%` vs. `66.77%`), `+15.55` points on `When2Call` (`81.13%` vs. `65.58%`), and `+6.80` points on `PhishNChips` (`78.75%` vs. `71.95%`). Across all model scales up to 27.4B, FastBrain-9B ranks **2nd of 7** overall, surpassing `27B Jev` (`80.32%`) and `26B-A4B DiffusionGemma Jev` (`69.14%`), and exceeding `27.4B Clef` (`81.81%`) on 5 of 10 individual benchmarks (`BFCL`, `API-Bank`, `CLINC150+OOS`, `Home Appliances`, and `When2Call`).
|
| 160 |
|
|
|
|
| 167 |

|
| 168 |
*Figure 3. Confidence calibration, open-set out-of-scope rejection (`CLINC150+OOS`), and abstention margin analysis (`When2Call`).*
|
| 169 |
|
| 170 |
+
**Table 4: Per-benchmark results on Cloudflare Decision Index (`v0.2.1`, 10 tasks, \\(N = 22,427\\)). Bold denotes the highest score across all parameter scales; underline denotes best in the ≤9B class.**
|
| 171 |
|
| 172 |
| Benchmark (Metric) | **FastBrain-9B** (Ours) | Clef-flash (Base) | Kev 9B | Clef | Jev | DiffGemma | Laya |
|
| 173 |
| :--- | :---: | :---: | :---: | :---: | :---: | :---: | :---: |
|
|
|
|
| 190 |
|
| 191 |
### 4.1 Runtime Comparison Across Inference Engines
|
| 192 |
|
| 193 |
+
Table 5 compares end-to-end latency, prefill throughput, and memory footprint across three serving backends—SGLang (`lmsysorg/sglang:dev-clef` with `JointSchemaPooler`), `llama.cpp` (`b11430` `ClefModel` graph in `Q8_0` and `Q4_K_M`), and native PyTorch (`joint_schema_model.py`)—benchmarking identical requests on a single NVIDIA RTX PRO 6000 Blackwell Server Edition GPU. Because `output_tokens = 0` for all requests, throughput is governed purely by prefill compute and span-pooling efficiency, reaching `19,319 tok/s` (`131.2` decisions/s) at batch size \\(B = 4\\) in SGLang.
|
| 194 |
|
| 195 |
**Table 5: End-to-end latency and prefill throughput on NVIDIA RTX PRO 6000 Blackwell Server Edition (G4).**
|
| 196 |
|
|
|
|
| 200 |
| Peak GPU Memory (16K context) | ~20.5 GiB | 12.55 GiB | **10.30 GiB** | 20.3–24.7 GiB |
|
| 201 |
| Compatible with 16 GiB T4 GPU | No (BF16 TP=1) | Yes (`9.03 GiB` at 4K) | **Yes (`6.78 GiB` at 4K)** | Yes (`6.8 GiB` via NF4) |
|
| 202 |
| Top-1 Agreement vs. PyTorch BF16 | **100.00%** | 99.86% (1,471 / 1,473) | 98.78% (1,455 / 1,473) | 100.00% (reference) |
|
| 203 |
+
| 1. Short Intent Routing (293 tok, \\(K=2\\)) | **30.01 ms** (9,763 tok/s) | 32.82 ms (8,929 tok/s) | 31.69 ms (9,245 tok/s) | 43.23 ms (6,778 tok/s) |
|
| 204 |
+
| 2. Typed Decisions (736 tok, \\(K=5\\)) | **50.29 ms** (14,635 tok/s) | 62.40 ms (11,795 tok/s) | 61.80 ms (11,909 tok/s) | 63.58 ms (11,576 tok/s) |
|
| 205 |
+
| 3. When2Call Governance (1,068 tok, \\(K=1\\)) | **66.06 ms** (16,167 tok/s) | 87.78 ms (12,167 tok/s) | 89.34 ms (11,955 tok/s) | 77.77 ms (13,733 tok/s) |
|
| 206 |
+
| 4. PhishNChips Security (1,303 tok, \\(K=9\\)) | **75.86 ms** (17,176 tok/s) | 108.55 ms (12,004 tok/s) | 109.97 ms (11,849 tok/s) | 90.32 ms (14,426 tok/s) |
|
| 207 |
+
| 5. SafeAgents Judge (2,217 tok, \\(K=2\\)) | **125.85 ms** (17,616 tok/s) | 181.70 ms (12,201 tok/s) | 183.50 ms (12,082 tok/s) | 138.06 ms (16,058 tok/s) |
|
| 208 |
+
| Batch \\(B=4\\) Typed Decisions (2,944 tok, 20 dec) | **152.39 ms** (19,319 tok/s) | 248.50 ms (11,847 tok/s) | 245.20 ms (12,006 tok/s) | 175.85 ms (16,742 tok/s) |
|
| 209 |
|
| 210 |
### 4.2 Mixed-Precision GGUF Quantization Fidelity
|
| 211 |
|
|
|
|
| 215 |
|
| 216 |
| Metric / Subset | PyTorch `bfloat16` (18.82 GB) | GGUF `Q8_0` (9.66 GB) | GGUF `Q4_K_M` (6.73 GB, `Q8_0` Head) |
|
| 217 |
| :--- | :---: | :---: | :---: |
|
| 218 |
+
| Top-1 Decision Agreement (\\(N = 1,473\\)) | 100.00% (ref) | 99.86% (1,471 / 1,473) | 98.78% (1,455 / 1,473) |
|
| 219 |
+
| Mean Absolute Probability Error (\\(\lvert \hat{p}_q - \hat{p}_{\mathrm{bf16}} \rvert\\)) | 0.00000 | 0.00203 | 0.00387 |
|
| 220 |
| 10-Task Mean (Uncalibrated Head) | 82.37% | 82.34% (-0.03) | 82.56% (+0.19) |
|
| 221 |
| 10-Task Mean (Schema-Calibrated) | 83.68% | 83.65% (-0.03) | 83.48% (-0.20) |
|
| 222 |
| ├─ 1. BFCL / 3. API-Bank / 4. BANKING77 | 100.0% / 95.0% / 100.0% | 100.0% / 95.0% / 100.0% | 100.0% / 95.0% / 100.0% |
|
README_zh.md
CHANGED
|
@@ -39,16 +39,16 @@ FastBrain-9B 主干网络包含 32 层混合序列建模模块(`9.30B` 参数
|
|
| 39 |
|
| 40 |
### 2.2 联合 Schema 决策头与向量化 CUDA 算子
|
| 41 |
|
| 42 |
-
`JointSchemaHead`(`116M` 参数,独立存储于 `joint_head.safetensors`)从主干末层隐藏状态 \(H \in \mathbb{R}^{T \times d}\) 中提取由 `<|question_start|>`/`<|question_end|>` 与 `<|option_start|>`/`<|option_end|>` 界定的片段表征,经问题-选项交叉注意力交互后映射为三类基础决策原语的标量对数几率:
|
| 43 |
-
1. **`choice`**(\(K\) 分类离散分布):输出归一化概率向量 \(\hat{p} = \operatorname{softmax}(z / \tau) \in \Delta^{K-1}\)。
|
| 44 |
-
2. **`noul`**(二元真值判别):输出命题成立概率 \(\hat{p}(\text{true}) = \sigma(z_{\text{true}} - z_{\text{false}})\)。
|
| 45 |
-
3. **`score`**(定义在有序分值 \(v_1 < \dots < v_K\) 上的量表评估):同时输出各等级概率 \(\hat{p} \in \Delta^{K-1}\) 与数学期望得分 \(\mathbb{E}[\hat{y}] = \sum_{k=1}^K \hat{p}_k v_k\)。
|
| 46 |
|
| 47 |
-
在参考运行时 `joint_schema_model.py` 中,针对 CUDA 设备实现了融合算子路径 `_forward_fast_cuda`:利用基于 `torch.cumsum` 的 \(\mathcal{O}(1)\) 前缀和差分替代逐片段 Python 循环完成变长片段均值池化,并通过 `cu_seqlens` 紧凑打包调用变长 FlashAttention(`torch.ops.aten._flash_attention_forward`)替代稠密掩码注意力。在 NVIDIA RTX PRO 6000 Blackwell 上,该实现将决策头单次前向耗时由 `3.27 ms` 降至 `1.72 ms`(**1.90 倍加速**),且保持严格数值等价(最大 Logit 绝对偏差 `max_logit_diff = 0.00000`)。
|
| 48 |
|
| 49 |
### 2.3 序数 KL 蒸馏与编译期 Schema 拓扑校准
|
| 50 |
|
| 51 |
-
标准交叉熵或普通多类 KL 散度将有序评定量表视为无序独立类别,对相邻等级误差与跨端极端误差施加同等惩罚。FastBrain-9B 在 `TypeScript/FastBrain-Instruct` 数据集上采用复合分布匹配损失进行训练,联合优化软标签 KL 散度、离散分级概率评分(预测累积分布 \(\hat{F}_k = \sum_{j \le k} \hat{p}_j\) 与目标累积分布 \(F_k\) 之间的二阶距离)以及 `score` 题型的期望值均方误差:
|
| 52 |
|
| 53 |
$$
|
| 54 |
\mathcal{L}(\theta) = \mathrm{KL}(p \,\|\, \hat{p}_\theta) + \lambda_{\mathrm{ord}} \sum_{k=1}^{K-1} \bigl(\hat{F}_{\theta,k} - F_k\bigr)^2 + \mathbb{I}[\text{type}=\text{score}] \cdot \lambda_{\mathrm{mse}} \bigl(\mathbb{E}[\hat{y}_\theta] - \mathbb{E}[y]\bigr)^2.
|
|
@@ -75,19 +75,19 @@ $$
|
|
| 75 |
|
| 76 |
### 3.1 智能体工具调用安全审计评测 (`safeagents-judge-bench`)
|
| 77 |
|
| 78 |
-
[`harness-hub/safeagents-judge-bench`](https://huggingface.co/datasets/harness-hub/safeagents-judge-bench/tree/v2) 用于评估决策模型对真实编码智能体(运行于 Terminal-Bench 2.1、OpenThoughts-TBLite 与 Terminal-Bench 4)在沙箱中被静态扫描器 `atif-scan` 标记的可疑工具调用的审计能力。`v2` 测试集包含来自 55 个任务的 \(N = 661\) 次工具调用(其中 `536` 次为分布在 `82` 条违规轨迹中的恶意 `block` 调用,`125` 次为正常 `allow` 调用)。每次调用以零样本单次请求形式同时询问一个二元 `noul` 拦截问题与一个 7 类 `choice` 违规归因问题(`legit`、`eval_gaming`、`harness_introspection`、`credential_misuse`、`exfiltration`、`sandbox_escape`、`other_overreach`)。表 2 中的 `v2` 官方结果由基准维护者在 NVIDIA A100-SXM4-80GB(HF Jobs)上独立运行并提交合入([PR #1](https://huggingface.co/TypeScript/FastBrain-9B/discussions/1))。
|
| 79 |
|
| 80 |
-
在 7 类违规归因赛道上,FastBrain-9B 取得 `0.751` 的 Macro-F1 与 `0.858` 的分类准确率(`567/661`),位列全部参评模型第 1:相较于参数量约为其 3 倍的 27.4B 模型 `Cloudflare/clef`(Macro-F1 `0.563`,准确率 `0.737`),分别高出 `+18.8` 与 `+12.1` 个百分点;相较于同尺寸基座 `Cloudflare/clef-flash`(Macro-F1 `0.462`,准确率 `0.576`),分别提升 `+28.9` 与 `+28.2` 个百分点。分项指标上,`harness_introspection`(\(n=272\))F1 达 `0.945`(召回率 `0.978`),`eval_gaming`(\(n=245\))F1 达 `0.828`(召回率 `0.853`),合法调用 `legit`(\(n=125\))精确率达 `0.975`。
|
| 81 |
|
| 82 |
-
在二元安全拦截赛道上,FastBrain-9B 的 AUROC 为 `0.985`,恶意调用总召回率为 `0.996`(`534/536`),全部 `82/82` 条越界轨迹均在首次违规调用处即被捕获(`median_bad_calls_before_catch = 0.0`),其中 `harness_introspection`(`272/272`)、`credential_misuse`(`15/15`)与 `sandbox_escape`(`4/4`)三类召回率均为 `1.000`。在未经验证集调整的默认判别阈值 \(P(\text{block}) \ge 0.50\) 下,平衡准确率为 `0.814`(误拦率 `0.368`,优于 27.4B `Cloudflare/clef` 的平衡准确率 `0.808` 与误拦率 `0.384`)。由于连续概率排序区分度保持在 `0.985` AUROC,将工作点阈值设定为 \(P(\text{block}) \ge 0.961\) 时,平衡准确率提升至 `0.938`,误拦率降至 `0.056`(`7/125`),同时仍保持 `82/82` 越界轨迹全检出。表 2 同时列出了 `v1` 版本(\(N = 562\),78 条轨迹)的官方归档结果,FastBrain-9B 在该版本二元与多分类双赛道均位居第 1 / 9。
|
| 83 |
|
| 84 |
-
**表 2:`harness-hub/safeagents-judge-bench` 零样本评测结果对比(`v2` 测试集 \(N = 661\),82 条越界轨迹;附 `v1` 归档对照 \(N = 562\))。**
|
| 85 |
|
| 86 |
| 模型 | 参数��� | 7 分类 Macro-F1 ↑ | 7 分类准确率 ↑ | AUROC ↑ | 平衡准确率 ↑ | 恶意召回率 ↑ | 误拦率 ↓ | 越界轨迹检出 ↑ | p50 延迟(测试硬件) |
|
| 87 |
| :--- | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: |
|
| 88 |
-
| **TypeScript/FastBrain-9B** (`v2`, 默认 \(t = 0.50\)) | 9.4B | **0.751** | **0.858** | 0.985 | 0.814 | 0.996 | 0.368 | **82 / 82** | 327 ms (A100) · 104 ms (G4) |
|
| 89 |
-
| **TypeScript/FastBrain-9B** (`v2`, 校准 \(t = 0.961\)) | 9.4B | **0.751** | **0.858** | 0.985 | **0.938** | 0.931 | 0.056 | **82 / 82** | 327 ms (A100) · 104 ms (G4) |
|
| 90 |
-
| **TypeScript/FastBrain-9B** (`v1` 官方榜, \(N = 562\)) | 9.4B | **0.850** | **0.923** | 0.983 | **0.974** | **1.000** | 0.051 | **78 / 78** | 104 ms (RTX PRO 6000) |
|
| 91 |
| Cloudflare/clef | 27.4B | 0.563 | 0.737 | **0.993** | 0.808 | **1.000** | 0.384 | **82 / 82** | 636 ms (A100) |
|
| 92 |
| vllm-sr/d3-mini | 1.8B | 0.512 | 0.652 | 0.986 | 0.856 | **1.000** | 0.288 | **82 / 82** | 168 ms (A100) |
|
| 93 |
| Cloudflare/clef-flash (基座) | 9.4B | 0.462 | 0.576 | 0.967 | 0.893 | 0.978 | 0.192 | 81 / 82 | 326 ms (A100) |
|
|
@@ -102,11 +102,11 @@ $$
|
|
| 102 |
|
| 103 |
[`LocalLLaMA/typed-decisions`](https://huggingface.co/datasets/LocalLLaMA/typed-decisions) 评测集覆盖智能体轨迹可观测性(`agent_trace_observability`)、客户服务(`customer_service`)、供应商发票核验(`invoice_processing`)与安全事件响应(`security_incidents`)四类企业工作流,测试集包含 400 个独立案例、共 2,000 个类型化子问题(涵盖 `noul`、`choice` 与 `score`),并以多教师模型集成的软标签分布作为真值基准。官方仓库存在两种并列真值处理脚本:`bench/score_with_jobs.py` 对真值分布中概率并列最高的选项按首个索引判对(`g.index(max(g))`),而 `scripts/score_model.py` 将 20 道并列题中的任一最高分选项均计为正确;两种脚本下的 KL 散度与 Brier 评分完全一致。我们在表 3 中同时报告两种口径的准确率,并在 [`eval_submissions/`](./eval_submissions/) 中公开原始预测文件。
|
| 104 |
|
| 105 |
-
在零样本总榜(\(N = 43\) 个参评模型)中,FastBrain-9B 取得 `0.7525`(并列口径 `0.7625`)的准确率、`0.1508` 的 KL 散度、`0.0795` 的 Brier 评分与 `0.085` 的期望校准误差(ECE),在 RTX PRO 6000(SGLang)上的 p50 延迟为 `50 ms`。该成绩位列全量零样本榜 **第 7 / 43**、开源权重模型 **第 5 / 36**、≤9B 开源模型 **第 2 / 19** 以及全部 `Clef` 架构模型 **第 1**,较 9.4B 基座 `Cloudflare/clef-flash`(准确率 `0.7100`,KL `0.2093`)提升 `+4.25` 个百分点,较 27.4B `Cloudflare/clef`(准确率 `0.7275`,KL `0.1907`)提升 `+2.50` 个百分点。在零样本准确率高于 `0.700` 的全部 28 个模型中,FastBrain-9B 具有最低的 p50 推理延迟(`50 ms`,对比 9B `Amos 1 medium` 的 `118 ms`、27B `Amos 1 large` 的 `357 ms`、`meraGPT Decider 1` 的 `526 ms` 以及 31B `Quyet-1.0-Large` 的 `3,335 ms`)。若直接评测未经后验规则调整的序数 KL 蒸馏原生头,零样本准确率为 `0.7735`(KL `0.1039`,Brier `0.0582`)。
|
| 106 |
|
| 107 |
-
在使用 `train` 划分(1,200 个样本)的独立榜单(\(N = 51\) 个模型)上,在**完全冻结主干与决策头权重(`no weights changed`)**、仅拟合逐问题对数先验偏置与标量温度参数的前提下,FastBrain-9B 取得 `0.7970`(并列口径 `0.8070`)的准确率、`0.0816` 的 KL 散度与 `0.0437` 的 Brier 评分,位列 **并列第 12(列表第 14 / 51)**,并在冻结权重模型中取得最优的 KL 分布校准度。
|
| 108 |
|
| 109 |
-
**表 3:`LocalLLaMA/typed-decisions` 测试集(\(N = 400\) 案例,2,000 次决策)评测结果。`/` 前后分别对应 `score_with_jobs.py` 与 `score_model.py` 口径。**
|
| 110 |
|
| 111 |
| 模型 | 参数量 | 评测设定 | 总体准确率 ↑ | KL 散度 ↓ | Brier ↓ | `noul` ↑ | `choice` ↑ | `score` ↑ | Agent ↑ | 客服 ↑ | 发票 ↑ | 安全 ↑ | p50 延迟 |
|
| 112 |
| :--- | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: |
|
|
@@ -128,7 +128,7 @@ $$
|
|
| 128 |
|
| 129 |
### 3.3 多领域结构化决策基准 (Cloudflare Decision Index `v0.2.1`)
|
| 130 |
|
| 131 |
-
在涵盖 10 个评测数据集、共 \(N = 22,427\) 条请求的 **Cloudflare Decision Index(`v0.2.1`)** 套件(覆盖 AST 函数调用、20 候选工具检索重排、多步状态 API 调用、细粒度金融意图识别、开放集越界拒识、推理密集型检索、电商相关性分级、多探针钓鱼网站鉴别与工具调用触发治理)上,FastBrain-9B 的无加权 10 任务宏平均得分为 **80.55%**(表 4,图 1–3)。
|
| 132 |
|
| 133 |
在 ≤9B 参数量级(对比 `Clef-flash` 9.4B、`Kev 9B` 与 `Laya` 0.4B)中,FastBrain-9B 在全部 **10 / 10 个子基准上均取得第 1**,宏平均分较同架构基座 `Clef-flash`(`74.96%`)提升 `+5.59` 个百分点,较 `Kev 9B`(`76.51%`)提升 `+4.04` 个百分点。增幅主要集中于对边界校准敏感的开放集拒识与安全治理任务:`CLINC150+OOS` 提升 `+31.67` 个百分点(`98.44%` 对比 `66.77%`),`When2Call` 提升 `+15.55` 个百分点(`81.13%` 对比 `65.58%`),`PhishNChips` 提升 `+6.80` 个百分点(`78.75%` 对比 `71.95%`)。跨全部参数规模(最高 27.4B)比较,FastBrain-9B 总宏平均位列 **第 2 / 7**,高于 `27B Jev`(`80.32%`)与 `26B-A4B DiffusionGemma Jev`(`69.14%`),并在 `BFCL`、`API-Bank`、`CLINC150+OOS`、`Home Appliances` 与 `When2Call` 五个子基准上高于 `27.4B Clef`(`81.81%`)。
|
| 134 |
|
|
@@ -141,7 +141,7 @@ $$
|
|
| 141 |

|
| 142 |
*图 3. 置信度可靠性图、开放集越界意图拒识(`CLINC150+OOS`)与工具触发边际分析(`When2Call`)。*
|
| 143 |
|
| 144 |
-
**表 4:Cloudflare Decision Index(`v0.2.1`,10 任务,\(N = 22,427\))逐项评测结果。粗体表示全参数规模最高值,下划线表示 ≤9B 量级最高值。**
|
| 145 |
|
| 146 |
| 评测基准(指标) | **FastBrain-9B** (本工作) | Clef-flash (基座) | Kev 9B | Clef | Jev | DiffGemma | Laya |
|
| 147 |
| :--- | :---: | :---: | :---: | :---: | :---: | :---: | :---: |
|
|
@@ -164,7 +164,7 @@ $$
|
|
| 164 |
|
| 165 |
### 4.1 不同推理引擎性能对比
|
| 166 |
|
| 167 |
-
表 5 展示了在同一张 NVIDIA RTX PRO 6000 Blackwell Server Edition GPU 上,SGLang(集成 `JointSchemaPooler` 的 `lmsysorg/sglang:dev-clef`)、`llama.cpp`(`b11430` `ClefModel` 计算图,含 `Q8_0` 与 `Q4_K_M` 两种精度)以及原生 PyTorch(`joint_schema_model.py`)在五类典型长度负载下的端到端延迟、Prefill 吞吐率与显存占
|
| 168 |
|
| 169 |
**表 5:NVIDIA RTX PRO 6000 Blackwell Server Edition(G4)上三类推理后端的延迟与吞吐实测。**
|
| 170 |
|
|
@@ -174,23 +174,23 @@ $$
|
|
| 174 |
| 峰值显存占用 (16K 上下文) | ~20.5 GiB | 12.55 GiB | **10.30 GiB** | 20.3–24.7 GiB |
|
| 175 |
| 兼容 16 GiB T4 显卡 | 否 (BF16 TP=1) | 是 (`9.03 GiB`, 4K) | **是 (`6.78 GiB`, 4K)** | 是 (`6.8 GiB`, NF4) |
|
| 176 |
| 相对 PyTorch BF16 Top-1 一致率 | **100.00%** | 99.86% (1,471 / 1,473) | 98.78% (1,455 / 1,473) | 100.00% (基准) |
|
| 177 |
-
| 1. 短文本意图路由 (293 tok, \(K=2\)) | **30.01 ms** (9,763 tok/s) | 32.82 ms (8,929 tok/s) | 31.69 ms (9,245 tok/s) | 43.23 ms (6,778 tok/s) |
|
| 178 |
-
| 2. Typed Decisions (736 tok, \(K=5\)) | **50.29 ms** (14,635 tok/s) | 62.40 ms (11,795 tok/s) | 61.80 ms (11,909 tok/s) | 63.58 ms (11,576 tok/s) |
|
| 179 |
-
| 3. When2Call 工具治理 (1,068 tok, \(K=1\)) | **66.06 ms** (16,167 tok/s) | 87.78 ms (12,167 tok/s) | 89.34 ms (11,955 tok/s) | 77.77 ms (13,733 tok/s) |
|
| 180 |
-
| 4. PhishNChips 安全检测 (1,303 tok, \(K=9\)) | **75.86 ms** (17,176 tok/s) | 108.55 ms (12,004 tok/s) | 109.97 ms (11,849 tok/s) | 90.32 ms (14,426 tok/s) |
|
| 181 |
-
| 5. SafeAgents 审计 (2,217 tok, \(K=2\)) | **125.85 ms** (17,616 tok/s) | 181.70 ms (12,201 tok/s) | 183.50 ms (12,082 tok/s) | 138.06 ms (16,058 tok/s) |
|
| 182 |
-
| 批量 \(B=4\) Typed Decisions (2,944 tok, 20 题) | **152.39 ms** (19,319 tok/s) | 248.50 ms (11,847 tok/s) | 245.20 ms (12,006 tok/s) | 175.85 ms (16,742 tok/s) |
|
| 183 |
|
| 184 |
### 4.2 混合精度 GGUF 量化保真度分析
|
| 185 |
|
| 186 |
若对交叉注意力决策头或线性注意力递归状态直接施加均匀 4-bit 量化,细粒度选项间微小的对数几率差值易受量化噪声干扰。因此在 `FastBrain-9B-Q4_K_M.gguf`(`6.73 GB`)构建方案中,仅对前馈网络(FFN)使用 4-bit K-quants,而将全部 `dec.*` 决策头张量、`ssm_*` Gated DeltaNet 状态参数以及 `attn_*` 投影矩阵保持为 `Q8_0` 精度。在涵盖 10 类任务的分层验证子集(`203` 个多约束请求,共 `1,473` 道独立评分题;表 6)上,`Q8_0.gguf`(`9.66 GB`)在 `1,471/1,473` 道题上与原生 `bfloat16` 给出完全一致的 Top-1 决策(一致率 `99.86%`,预测概率平均绝对误差 `0.00203`,10 任务均分差异 `-0.03` 个百分点);`Q4_K_M.gguf` 保持了 `98.78%` 的 Top-1 决策一致率(概率 MAE `0.00387`,校准后均分差异 `-0.20` 个百分点)。表 7 给出了各精度版本在不同硬件(含 NVIDIA Tesla T4 `16 GiB` 实测)上的资源需求。
|
| 187 |
|
| 188 |
-
**表 6:10 任务分层子集(\(N = 1,473\) 道评分题)上 GGUF 量化模型的决策一致性与精度保持率。**
|
| 189 |
|
| 190 |
| 评测指标 / 子任务 | PyTorch `bfloat16` (18.82 GB) | GGUF `Q8_0` (9.66 GB) | GGUF `Q4_K_M` (6.73 GB, `Q8_0` 决策头) |
|
| 191 |
| :--- | :---: | :---: | :---: |
|
| 192 |
-
| Top-1 决策一致率 (\(N = 1,473\)) | 100.00% (基准) | 99.86% (1,471 / 1,473) | 98.78% (1,455 / 1,473) |
|
| 193 |
-
| 预测概率平均绝对误差 (\(\lvert \hat{p}_q - \hat{p}_{\mathrm{bf16}} \rvert\)) | 0.00000 | 0.00203 | 0.00387 |
|
| 194 |
| 10 任务宏平均(未校准原生输出) | 82.37% | 82.34% (-0.03) | 82.56% (+0.19) |
|
| 195 |
| 10 任务宏平均(Schema 拓扑校准) | 83.68% | 83.65% (-0.03) | 83.48% (-0.20) |
|
| 196 |
| ├─ 1. BFCL / 3. API-Bank / 4. BANKING77 | 100.0% / 95.0% / 100.0% | 100.0% / 95.0% / 100.0% | 100.0% / 95.0% / 100.0% |
|
|
|
|
| 39 |
|
| 40 |
### 2.2 联合 Schema 决策头与向量化 CUDA 算子
|
| 41 |
|
| 42 |
+
`JointSchemaHead`(`116M` 参数,独立存储于 `joint_head.safetensors`)从主干末层隐藏状态 \\(H \in \mathbb{R}^{T \times d}\\) 中提取由 `<|question_start|>`/`<|question_end|>` 与 `<|option_start|>`/`<|option_end|>` 界定的片段表征,经问题-选项交叉注意力交互后映射为三类基础决策原语的标量对数几率:
|
| 43 |
+
1. **`choice`**(\\(K\\) 分类离散分布):输出归一化概率向量 \\(\hat{p} = \operatorname{softmax}(z / \tau) \in \Delta^{K-1}\\)。
|
| 44 |
+
2. **`noul`**(二元真值判别):输出命题成立概率 \\(\hat{p}(\text{true}) = \sigma(z_{\text{true}} - z_{\text{false}})\\)。
|
| 45 |
+
3. **`score`**(定义在有序分值 \\(v_1 < \dots < v_K\\) 上的量表评估):同时输出各等级概率 \\(\hat{p} \in \Delta^{K-1}\\) 与数学期望得分 \\(\mathbb{E}[\hat{y}] = \sum_{k=1}^K \hat{p}_k v_k\\)。
|
| 46 |
|
| 47 |
+
在参考运行时 `joint_schema_model.py` 中,针对 CUDA 设备实现了融合算子路径 `_forward_fast_cuda`:利用基于 `torch.cumsum` 的 \\(\mathcal{O}(1)\\) 前缀和差分替代逐片段 Python 循环完成变长片段均值池化,并通过 `cu_seqlens` 紧凑打包调用变长 FlashAttention(`torch.ops.aten._flash_attention_forward`)替代稠密掩码注意力。在 NVIDIA RTX PRO 6000 Blackwell 上,该实现将决策头单次前向耗时由 `3.27 ms` 降至 `1.72 ms`(**1.90 倍加速**),且保持严格数值等价(最大 Logit 绝对偏差 `max_logit_diff = 0.00000`)。
|
| 48 |
|
| 49 |
### 2.3 序数 KL 蒸馏与编译期 Schema 拓扑校准
|
| 50 |
|
| 51 |
+
标准交叉熵或普通多类 KL 散度将有序评定量表视为无序独立类别,对相邻等级误差与跨端极端误差施加同等惩罚。FastBrain-9B 在 `TypeScript/FastBrain-Instruct` 数据集上采用复合分布匹配损失进行训练,联合优化软标签 KL 散度、离散分级概率评分(预测累积分布 \\(\hat{F}_k = \sum_{j \le k} \hat{p}_j\\) 与目标累积分布 \\(F_k\\) 之间的二阶距离)以及 `score` 题型的期望值均方误差:
|
| 52 |
|
| 53 |
$$
|
| 54 |
\mathcal{L}(\theta) = \mathrm{KL}(p \,\|\, \hat{p}_\theta) + \lambda_{\mathrm{ord}} \sum_{k=1}^{K-1} \bigl(\hat{F}_{\theta,k} - F_k\bigr)^2 + \mathbb{I}[\text{type}=\text{score}] \cdot \lambda_{\mathrm{mse}} \bigl(\mathbb{E}[\hat{y}_\theta] - \mathbb{E}[y]\bigr)^2.
|
|
|
|
| 75 |
|
| 76 |
### 3.1 智能体工具调用安全审计评测 (`safeagents-judge-bench`)
|
| 77 |
|
| 78 |
+
[`harness-hub/safeagents-judge-bench`](https://huggingface.co/datasets/harness-hub/safeagents-judge-bench/tree/v2) 用于评估决策模型对真实编码智能体(运行于 Terminal-Bench 2.1、OpenThoughts-TBLite 与 Terminal-Bench 4)在沙箱中被静态扫描器 `atif-scan` 标记的可疑工具调用的审计能力。`v2` 测试集包含来自 55 个任务的 \\(N = 661\\) 次工具调用(其中 `536` 次为分布在 `82` 条违规轨迹中的恶意 `block` 调用,`125` 次为正常 `allow` 调用)。每次调用以零样本单次请求形式同时询问一个二元 `noul` 拦截问题与一个 7 类 `choice` 违规归因问题(`legit`、`eval_gaming`、`harness_introspection`、`credential_misuse`、`exfiltration`、`sandbox_escape`、`other_overreach`)。表 2 中的 `v2` 官方结果由基准维护者在 NVIDIA A100-SXM4-80GB(HF Jobs)上独立运行并提交合入([PR #1](https://huggingface.co/TypeScript/FastBrain-9B/discussions/1))。
|
| 79 |
|
| 80 |
+
在 7 类违规归因赛道上,FastBrain-9B 取得 `0.751` 的 Macro-F1 与 `0.858` 的分类准确率(`567/661`),位列全部参评模型第 1:相较于参数量约为其 3 倍的 27.4B 模型 `Cloudflare/clef`(Macro-F1 `0.563`,准确率 `0.737`),分别高出 `+18.8` 与 `+12.1` 个百分点;相较于同尺寸基座 `Cloudflare/clef-flash`(Macro-F1 `0.462`,准确率 `0.576`),分别提升 `+28.9` 与 `+28.2` 个百分点。分项指标上,`harness_introspection`(\\(n=272\\))F1 达 `0.945`(召回率 `0.978`),`eval_gaming`(\\(n=245\\))F1 达 `0.828`(召回率 `0.853`),合法调用 `legit`(\\(n=125\\))精确率达 `0.975`。
|
| 81 |
|
| 82 |
+
在二元安全拦截赛道上,FastBrain-9B 的 AUROC 为 `0.985`,恶意调用总召回率为 `0.996`(`534/536`),全部 `82/82` 条越界轨迹均在首次违规调用处即被捕获(`median_bad_calls_before_catch = 0.0`),其中 `harness_introspection`(`272/272`)、`credential_misuse`(`15/15`)与 `sandbox_escape`(`4/4`)三类召回率均为 `1.000`。在未经验证集调整的默认判别阈值 \\(P(\text{block}) \ge 0.50\\) 下,平衡准确率为 `0.814`(误拦率 `0.368`,优于 27.4B `Cloudflare/clef` 的平衡准确率 `0.808` 与误拦率 `0.384`)。由于连续概率排序区分度保持在 `0.985` AUROC,将工作点阈值设定为 \\(P(\text{block}) \ge 0.961\\) 时,平衡准确率提升至 `0.938`,误拦率降至 `0.056`(`7/125`),同时仍保持 `82/82` 越界轨迹全检出。表 2 同时列出了 `v1` 版本(\\(N = 562\\),78 条轨迹)的官方归档结果,FastBrain-9B 在该版本二元与多分类双赛道均位居第 1 / 9。
|
| 83 |
|
| 84 |
+
**表 2:`harness-hub/safeagents-judge-bench` 零样本评测结果对比(`v2` 测试集 \\(N = 661\\),82 条越界轨迹;附 `v1` 归档对照 \\(N = 562\\))。**
|
| 85 |
|
| 86 |
| 模型 | 参数��� | 7 分类 Macro-F1 ↑ | 7 分类准确率 ↑ | AUROC ↑ | 平衡准确率 ↑ | 恶意召回率 ↑ | 误拦率 ↓ | 越界轨迹检出 ↑ | p50 延迟(测试硬件) |
|
| 87 |
| :--- | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: |
|
| 88 |
+
| **TypeScript/FastBrain-9B** (`v2`, 默认 \\(t = 0.50\\)) | 9.4B | **0.751** | **0.858** | 0.985 | 0.814 | 0.996 | 0.368 | **82 / 82** | 327 ms (A100) · 104 ms (G4) |
|
| 89 |
+
| **TypeScript/FastBrain-9B** (`v2`, 校准 \\(t = 0.961\\)) | 9.4B | **0.751** | **0.858** | 0.985 | **0.938** | 0.931 | 0.056 | **82 / 82** | 327 ms (A100) · 104 ms (G4) |
|
| 90 |
+
| **TypeScript/FastBrain-9B** (`v1` 官方榜, \\(N = 562\\)) | 9.4B | **0.850** | **0.923** | 0.983 | **0.974** | **1.000** | 0.051 | **78 / 78** | 104 ms (RTX PRO 6000) |
|
| 91 |
| Cloudflare/clef | 27.4B | 0.563 | 0.737 | **0.993** | 0.808 | **1.000** | 0.384 | **82 / 82** | 636 ms (A100) |
|
| 92 |
| vllm-sr/d3-mini | 1.8B | 0.512 | 0.652 | 0.986 | 0.856 | **1.000** | 0.288 | **82 / 82** | 168 ms (A100) |
|
| 93 |
| Cloudflare/clef-flash (基座) | 9.4B | 0.462 | 0.576 | 0.967 | 0.893 | 0.978 | 0.192 | 81 / 82 | 326 ms (A100) |
|
|
|
|
| 102 |
|
| 103 |
[`LocalLLaMA/typed-decisions`](https://huggingface.co/datasets/LocalLLaMA/typed-decisions) 评测集覆盖智能体轨迹可观测性(`agent_trace_observability`)、客户服务(`customer_service`)、供应商发票核验(`invoice_processing`)与安全事件响应(`security_incidents`)四类企业工作流,测试集包含 400 个独立案例、共 2,000 个类型化子问题(涵盖 `noul`、`choice` 与 `score`),并以多教师模型集成的软标签分布作为真值基准。官方仓库存在两种并列真值处理脚本:`bench/score_with_jobs.py` 对真值分布中概率并列最高的选项按首个索引判对(`g.index(max(g))`),而 `scripts/score_model.py` 将 20 道并列题中的任一最高分选项均计为正确;两种脚本下的 KL 散度与 Brier 评分完全一致。我们在表 3 中同时报告两种口径的准确率,并在 [`eval_submissions/`](./eval_submissions/) 中公开原始预测文件。
|
| 104 |
|
| 105 |
+
在零样本总榜(\\(N = 43\\) 个参评模型)中,FastBrain-9B 取得 `0.7525`(并列口径 `0.7625`)的准确率、`0.1508` 的 KL 散度、`0.0795` 的 Brier 评分与 `0.085` 的期望校准误差(ECE),在 RTX PRO 6000(SGLang)上的 p50 延迟为 `50 ms`。该成绩位列全量零样本榜 **第 7 / 43**、开源权重模型 **第 5 / 36**、≤9B 开源模型 **第 2 / 19** 以及全部 `Clef` 架构模型 **第 1**,较 9.4B 基座 `Cloudflare/clef-flash`(准确率 `0.7100`,KL `0.2093`)提升 `+4.25` 个百分点,较 27.4B `Cloudflare/clef`(准确率 `0.7275`,KL `0.1907`)提升 `+2.50` 个百分点。在零样本准确率高于 `0.700` 的全部 28 个模型中,FastBrain-9B 具有最低的 p50 推理延迟(`50 ms`,对比 9B `Amos 1 medium` 的 `118 ms`、27B `Amos 1 large` 的 `357 ms`、`meraGPT Decider 1` 的 `526 ms` 以及 31B `Quyet-1.0-Large` 的 `3,335 ms`)。若直接评测未经后验规则调整的序数 KL 蒸馏原生头,零样本准确率为 `0.7735`(KL `0.1039`,Brier `0.0582`)。
|
| 106 |
|
| 107 |
+
在使用 `train` 划分(1,200 个样本)的独立榜单(\\(N = 51\\) 个模型)上,在**完全冻结主干与决策头权重(`no weights changed`)**、仅拟合逐问题对数先验偏置与标量温度参数的前提下,FastBrain-9B 取得 `0.7970`(并列口径 `0.8070`)的准确率、`0.0816` 的 KL 散度与 `0.0437` 的 Brier 评分,位列 **并列第 12(列表第 14 / 51)**,并在冻结权重模型中取得最优的 KL 分布校准度。
|
| 108 |
|
| 109 |
+
**表 3:`LocalLLaMA/typed-decisions` 测试集(\\(N = 400\\) 案例,2,000 次决策)评测结果。`/` 前后分别对应 `score_with_jobs.py` 与 `score_model.py` 口径。**
|
| 110 |
|
| 111 |
| 模型 | 参数量 | 评测设定 | 总体准确率 ↑ | KL 散度 ↓ | Brier ↓ | `noul` ↑ | `choice` ↑ | `score` ↑ | Agent ↑ | 客服 ↑ | 发票 ↑ | 安全 ↑ | p50 延迟 |
|
| 112 |
| :--- | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: |
|
|
|
|
| 128 |
|
| 129 |
### 3.3 多领域结构化决策基准 (Cloudflare Decision Index `v0.2.1`)
|
| 130 |
|
| 131 |
+
在涵盖 10 个评测数据集、共 \\(N = 22,427\\) 条请求的 **Cloudflare Decision Index(`v0.2.1`)** 套件(覆盖 AST 函数调用、20 候选工具检索重排、多步状态 API 调用、细粒度金融意图识别、开放集越界拒识、推理密集型检索、电商相关性分级、多探针钓鱼网站鉴别与工具调用触发治理)上,FastBrain-9B 的无加权 10 任务宏平均得分为 **80.55%**(表 4,图 1–3)。
|
| 132 |
|
| 133 |
在 ≤9B 参数量级(对比 `Clef-flash` 9.4B、`Kev 9B` 与 `Laya` 0.4B)中,FastBrain-9B 在全部 **10 / 10 个子基准上均取得第 1**,宏平均分较同架构基座 `Clef-flash`(`74.96%`)提升 `+5.59` 个百分点,较 `Kev 9B`(`76.51%`)提升 `+4.04` 个百分点。增幅主要集中于对边界校准敏感的开放集拒识与安全治理任务:`CLINC150+OOS` 提升 `+31.67` 个百分点(`98.44%` 对比 `66.77%`),`When2Call` 提升 `+15.55` 个百分点(`81.13%` 对比 `65.58%`),`PhishNChips` 提升 `+6.80` 个百分点(`78.75%` 对比 `71.95%`)。跨全部参数规模(最高 27.4B)比较,FastBrain-9B 总宏平均位列 **第 2 / 7**,高于 `27B Jev`(`80.32%`)与 `26B-A4B DiffusionGemma Jev`(`69.14%`),并在 `BFCL`、`API-Bank`、`CLINC150+OOS`、`Home Appliances` 与 `When2Call` 五个子基准上高于 `27.4B Clef`(`81.81%`)。
|
| 134 |
|
|
|
|
| 141 |

|
| 142 |
*图 3. 置信度可靠性图、开放集越界意图拒识(`CLINC150+OOS`)与工具触发边际分析(`When2Call`)。*
|
| 143 |
|
| 144 |
+
**表 4:Cloudflare Decision Index(`v0.2.1`,10 任务,\\(N = 22,427\\))逐项评测结果。粗体表示全参数规模最高值,下划线表示 ≤9B 量级最高值。**
|
| 145 |
|
| 146 |
| 评测基准(指标) | **FastBrain-9B** (本工作) | Clef-flash (基座) | Kev 9B | Clef | Jev | DiffGemma | Laya |
|
| 147 |
| :--- | :---: | :---: | :---: | :---: | :---: | :---: | :---: |
|
|
|
|
| 164 |
|
| 165 |
### 4.1 不同推理引擎性能对比
|
| 166 |
|
| 167 |
+
表 5 展示了在同一张 NVIDIA RTX PRO 6000 Blackwell Server Edition GPU 上,SGLang(集成 `JointSchemaPooler` 的 `lmsysorg/sglang:dev-clef`)、`llama.cpp`(`b11430` `ClefModel` 计算图,含 `Q8_0` 与 `Q4_K_M` 两种精度)以及原生 PyTorch(`joint_schema_model.py`)在五类典型长度负载下的端到端延迟、Prefill 吞吐率与显存占���。由于全部请求的生成步数均为零(`output_tokens = 0`),推理耗时完全取决于 Prefill 计算与片段池化效率;在批量 \\(B=4\\) 并发设定下,SGLang 吞吐达 `19,319 tok/s`(每秒完成 `131.2` 个独立结构化决策)。
|
| 168 |
|
| 169 |
**表 5:NVIDIA RTX PRO 6000 Blackwell Server Edition(G4)上三类推理后端的延迟与吞吐实测。**
|
| 170 |
|
|
|
|
| 174 |
| 峰值显存占用 (16K 上下文) | ~20.5 GiB | 12.55 GiB | **10.30 GiB** | 20.3–24.7 GiB |
|
| 175 |
| 兼容 16 GiB T4 显卡 | 否 (BF16 TP=1) | 是 (`9.03 GiB`, 4K) | **是 (`6.78 GiB`, 4K)** | 是 (`6.8 GiB`, NF4) |
|
| 176 |
| 相对 PyTorch BF16 Top-1 一致率 | **100.00%** | 99.86% (1,471 / 1,473) | 98.78% (1,455 / 1,473) | 100.00% (基准) |
|
| 177 |
+
| 1. 短文本意图路由 (293 tok, \\(K=2\\)) | **30.01 ms** (9,763 tok/s) | 32.82 ms (8,929 tok/s) | 31.69 ms (9,245 tok/s) | 43.23 ms (6,778 tok/s) |
|
| 178 |
+
| 2. Typed Decisions (736 tok, \\(K=5\\)) | **50.29 ms** (14,635 tok/s) | 62.40 ms (11,795 tok/s) | 61.80 ms (11,909 tok/s) | 63.58 ms (11,576 tok/s) |
|
| 179 |
+
| 3. When2Call 工具治理 (1,068 tok, \\(K=1\\)) | **66.06 ms** (16,167 tok/s) | 87.78 ms (12,167 tok/s) | 89.34 ms (11,955 tok/s) | 77.77 ms (13,733 tok/s) |
|
| 180 |
+
| 4. PhishNChips 安全检测 (1,303 tok, \\(K=9\\)) | **75.86 ms** (17,176 tok/s) | 108.55 ms (12,004 tok/s) | 109.97 ms (11,849 tok/s) | 90.32 ms (14,426 tok/s) |
|
| 181 |
+
| 5. SafeAgents 审计 (2,217 tok, \\(K=2\\)) | **125.85 ms** (17,616 tok/s) | 181.70 ms (12,201 tok/s) | 183.50 ms (12,082 tok/s) | 138.06 ms (16,058 tok/s) |
|
| 182 |
+
| 批量 \\(B=4\\) Typed Decisions (2,944 tok, 20 题) | **152.39 ms** (19,319 tok/s) | 248.50 ms (11,847 tok/s) | 245.20 ms (12,006 tok/s) | 175.85 ms (16,742 tok/s) |
|
| 183 |
|
| 184 |
### 4.2 混合精度 GGUF 量化保真度分析
|
| 185 |
|
| 186 |
若对交叉注意力决策头或线性注意力递归状态直接施加均匀 4-bit 量化,细粒度选项间微小的对数几率差值易受量化噪声干扰。因此在 `FastBrain-9B-Q4_K_M.gguf`(`6.73 GB`)构建方案中,仅对前馈网络(FFN)使用 4-bit K-quants,而将全部 `dec.*` 决策头张量、`ssm_*` Gated DeltaNet 状态参数以及 `attn_*` 投影矩阵保持为 `Q8_0` 精度。在涵盖 10 类任务的分层验证子集(`203` 个多约束请求,共 `1,473` 道独立评分题;表 6)上,`Q8_0.gguf`(`9.66 GB`)在 `1,471/1,473` 道题上与原生 `bfloat16` 给出完全一致的 Top-1 决策(一致率 `99.86%`,预测概率平均绝对误差 `0.00203`,10 任务均分差异 `-0.03` 个百分点);`Q4_K_M.gguf` 保持了 `98.78%` 的 Top-1 决策一致率(概率 MAE `0.00387`,校准后均分差异 `-0.20` 个百分点)。表 7 给出了各精度版本在不同硬件(含 NVIDIA Tesla T4 `16 GiB` 实测)上的资源需求。
|
| 187 |
|
| 188 |
+
**表 6:10 任务分层子集(\\(N = 1,473\\) 道评分题)上 GGUF 量化模型的决策一致性与精度保持率。**
|
| 189 |
|
| 190 |
| 评测指标 / 子任务 | PyTorch `bfloat16` (18.82 GB) | GGUF `Q8_0` (9.66 GB) | GGUF `Q4_K_M` (6.73 GB, `Q8_0` 决策头) |
|
| 191 |
| :--- | :---: | :---: | :---: |
|
| 192 |
+
| Top-1 决策一致率 (\\(N = 1,473\\)) | 100.00% (基准) | 99.86% (1,471 / 1,473) | 98.78% (1,455 / 1,473) |
|
| 193 |
+
| 预测概率平均绝对误差 (\\(\lvert \hat{p}_q - \hat{p}_{\mathrm{bf16}} \rvert\\)) | 0.00000 | 0.00203 | 0.00387 |
|
| 194 |
| 10 任务宏平均(未校准原生输出) | 82.37% | 82.34% (-0.03) | 82.56% (+0.19) |
|
| 195 |
| 10 任务宏平均(Schema 拓扑校准) | 83.68% | 83.65% (-0.03) | 83.48% (-0.20) |
|
| 196 |
| ├─ 1. BFCL / 3. API-Bank / 4. BANKING77 | 100.0% / 95.0% / 100.0% | 100.0% / 95.0% / 100.0% | 100.0% / 95.0% / 100.0% |
|