Sajib Acharjee Dip's picture

Sajib Acharjee Dip

Sajib-006
Ā·

AI & ML interests

Multimodal Learning, Generative AI, Bioinformatics

Recent Activity

posted an update about 3 hours ago
šŸ›‘ Your agent's API said HTTP 200. The answer was still wrong. APIs fail silently. A misnamed filter gets ignored, a total gets capped at the page size, or an AND is read as OR, and the API still returns a normal-looking number. LLM agents trust it. We found this in 11 of 15 public research APIs, with counts inflated up to 691 times. Asking the model to double-check doesn't fix it. Checklists, reviewer models and LLM judges still leave over 70% of answers wrong. Even when a judge flags the bad count, agents report it anyway 77% of the time. So we moved the decision out of the model. SilentGuard is a harness that tests every tool result against simple invariants that any correct API must satisfy: count(A and B) + count(A and not B) == count(A) If a result breaks the rule twice, the harness withholds it, and the model never sees the wrong number. šŸ“Š Results on 7 open models: wrong answers fall from 74.0% to 1.7% on the faults the checks target, with no loss on clean tasks. On three frontier models they fall from 70.2% to 0.5%. šŸ¤— AgentOrchBench is a new benchmark of silent API failures with exact answers. It has count, list, retail and multi-step workflow tasks, plus 8 fault types you can switch on. from datasets import load_dataset tasks = load_dataset("Sajib-006/AgentOrchBench", "count", split="test") Dataset: https://huggingface.co/datasets/Sajib-006/AgentOrchBench Code (pip-installable, no dependencies): https://github.com/Sajib-006/silentguard Paper: https://arxiv.org/abs/2611.20595 Try the 30-second demo with no model and no API key: python examples/01_quickstart.py Joint work with Brian Ouyang and Lala Motlhabi (Genentech). Feedback welcome! šŸ™Œ
published a dataset 4 days ago
Sajib-006/AgentOrchBench
View all activity

Organizations

Hugging Face Discord Community's profile picture Fatima Institute Network's profile picture