AutoVend: Benchmarking LLM Agents on Long-Horizon Autonomous Vending Operations
Abstract
Large language model (LLM) agents are extending agent-mediated commerce on the Web from shopping for consumers to operating businesses for sellers. In online-to-offline (O2O) commerce, decisions are issued through Web-based platforms but realized through costly physical execution. Evaluating such agents requires going beyond individual transactions to ask whether decisions made online turn into sustained economic returns. We introduce AutoVend, a benchmark in which one LLM agent manages 20 vending machines across five site types through a Web-based operating interface for 180 simulated days. Its data-grounded simulator combines a shared warehouse, procurement lead times, paid replenishment, inventory expiration, and recurring operating costs. We evaluate five LLM agents against a rule-based inventory-control baseline and diagnose their site coordination, execution, and performance over time. Four of five evaluated LLM agents fall short of this baseline, and returns per dollar of inference cost rank models differently from returns alone. Operational analyses show that gains in individual metrics do not consistently yield stronger economic growth, as inventory and execution costs erode these gains. Trajectory analysis further reveals difficulties in grounding feasible actions, managing inventory commitments, and converting feedback into effective policy revisions.
Community
🤖 Can an LLM agent run a vending business for 180 days?
Many commercial tasks end when a transaction is completed. Running a business is different: today’s decisions determine tomorrow’s stock availability, costs, and cash position.
We introduce AutoVend, a benchmark where one LLM agent operates 20 vending machines across five site types for 180 simulated days through a digital operating interface.
📦 Digital decisions meet physical constraints
The agent chooses products, sets prices, orders inventory, and plans replenishment. A data-grounded simulator combines heterogeneous local demand with shared inventory and cash, procurement lead times, paid on-site replenishment, product expiration, and recurring operating costs. Stock in the warehouse cannot satisfy customers until it reaches machine shelves.
🔍 What does AutoVend test?
- Coordinating shared resources across sites with different demand.
- Planning procurement and replenishment under delays and costs.
- Using accumulated feedback to revise operating decisions over time.
đź“Š What do we find?
Four of five evaluated LLM agents fall short of a rule-based inventory-control baseline. Better individual operating metrics do not consistently translate into stronger economic returns, and accounting for inference costs changes model rankings.
AutoVend examines whether agents can turn repeated decisions into sustained business performance—and helps diagnose where that process breaks down.
niu bi
Get this paper in your agent:
hf papers read 2610.36980 Don't have the latest CLI?
curl -LsSf https://hf.co/cli/install.sh | bash Models citing this paper 0
No model linking this paper
Datasets citing this paper 0
No dataset linking this paper
Spaces citing this paper 0
No Space linking this paper
Collections including this paper 0
No Collection including this paper