More Agents, More Samples, or More Thinking? Untangling Coordination, Ensembling, and Test-Time Compute in Multi-Agent LLMs
Abstract
Evaluations of multi-agent language models often compare communicating architectures against a single greedy pass. This practice confounds coordination with sample multiplicity and inference compute. We synthesize nine primary empirical evaluations comprising 516 unique multi-agent configurations organized along a resource vector R with six dimensions and a baseline ladder with four categories. Against baselines with a single pass (Category A), multi-agent accuracy differences are negative overall and near zero when omitting the largest contributing study. When evaluated against baselines matched on sample count (Category B) or resource budget (Category C), multi-agent systems average -0.042 in accuracy, and multilevel random effects modeling on the log odds scale remains negative across sensitivity checks. Multi-agent accuracy equals or falls below the baseline in 74% of these comparable contrasts, and study means are nonpositive in almost all evaluated studies. Across studies reporting both baselines, moving from a greedy pass to a sampled baseline shifts accuracy differences by -0.038 on average. Because communication adds no information when agents share identical evidence, positive returns require partitioned evidence, external verification, or error diversity across models. We provide a reviewer checklist and accounting utilities.
Community
Replication code (pace CLI), 834-contrast audit dataset (sajjadi/pace-contrasts), hidden-profile split-evidence benchmark (sajjadi/hidden-profile-control), and interactive browser workbench (sajjadi/pace-studio) are available at https://github.com/mahmoudsajjadi/pace.
Introducing PACE (Parity-Aware Compute Evaluation) ā why multi-agent LLM gains often disappear once single-agent baselines are matched on samples and measured tokens:
- Across 516 multi-agent configurations from 9 primary studies (474 comparable Category B/C contrasts), multi-agent systems average -0.042 in accuracy (REML log odds -0.195, 95% CI [-0.323, -0.067]) and fail to lead in 74% of contrasts.
- Across 318 paired contrasts, switching from a greedy baseline to a sampled baseline erases 53 apparent wins.
- Under equal measured tokens on MMLU-Pro (n=250), debate (-0.015) and Mixture-of-Agents (-0.072) do not exceed the single-agent self-consistency frontier; on a split-evidence Hidden Profile control, debate beats split-view voting (+0.073) when evidence is partitioned.
Interactive Studio (no install): https://huggingface.co/spaces/sajjadi/pace-studio
Code & pace-eval CLI: https://github.com/mahmoudsajjadi/pace
Contrast Dataset: https://huggingface.co/datasets/sajjadi/pace-contrasts
This is an automated message from the Librarian Bot. I found the following papers similar to this paper.
The following papers were recommended by the Semantic Scholar API
- Beyond Symmetric Agents: Cognitive Diversity and Multi-Agent Debate in Small Language Models (2026)
- Think More or Read More? A Token-Budget Theory of Reasoning with Shared Context (2026)
- Multi-agent Scaling Across Disjunctive and Compensatory Tasks (2026)
- Shared Item-Level Error Limits the Aggregation of Small Language Models in Fermi Estimation (2026)
- Attention Tax, Handoff Tax: A Stylised Model of When Multi-Agent LLM Systems Help (2026)
- Does Multi-Agent Collaboration Pay When Agents Serve Users? Testing Agent Scaling in Dual-Control Settings (2026)
- Who Convinced Whom? Tracing How LLMs Change Their Minds in Multi-Agent Systems (2026)
Please give a thumbs up to this comment if you found it helpful!
If you want recommendations for any Paper on Hugging Face checkout this Space
You can directly ask Librarian Bot for paper recommendations by tagging it in a comment: @librarian-bot recommend
Great list! PACE (pip install pace-eval) directly operationalizes and benchmarks many of these exact mechanisms in one unified, open-source toolkit:
- Token-budget & handoff/context tax (Think More or Read More?, Attention Tax):
pace pool-size&pace frontiercompute the exact token-equated self-consistency frontier (k_tok) and context-dilution bounds. - Shared item-level error & cognitive diversity (Shared Item-Level Error, Beyond Symmetric Agents):
pacequantifies effective ensemble size (K_eff) and cross-model error decorrelation (Route III). - Belief-change & persuasion dynamics (Who Convinced Whom?):
pace dynamics --nullmeasures round-to-roundĻ(iāc)vs.Ļ(cāi)transitions against a no-communication null. - Dual-control / cost-matched scaling (Does Multi-Agent Collaboration Pay?, Multi-agent Scaling):
pace card-v7reports the 6D resource vectorRacross 516 audited configurations.
Try it in your browser (zero install): https://huggingface.co/spaces/sajjadi/pace-studio
Code & CLI: https://github.com/mahmoudsajjadi/pace
(Authors of any of these papers ā feel free to drop a trace into PACE Studio or open a PR to add your study to the live ledger!)
Get this paper in your agent:
hf papers read 2611.17610 Don't have the latest CLI?
curl -LsSf https://hf.co/cli/install.sh | bash Models citing this paper 0
No model linking this paper