AceAction-5M: Hierarchical Goals for Generalist Action Learning Across 3D Game Worlds
Abstract
AceAction provides multi-game human gameplay demonstrations annotated with task instructions, ordered subgoals, local skill instructions, and temporally aligned visual-action data to support learning skill composition and subgoal transitions in dynamic 3D environments.
Video games provide a practical setting for collecting vision-language-action (VLA) data, whose acquisition in physical environments is costly. Existing datasets, however, remain limited by restricted game coverage, sparse language supervision, or temporal hierarchies that do not necessarily reflect changes in task objectives. We introduce AceAction, a dataset of human gameplay across multiple 3D games for learning skill composition and subgoal transitions. AceAction organizes demonstrations around complete tasks, linking overall instructions, ordered subgoals, and local skill instructions to temporally aligned visual observations and keyboard-and-mouse actions. To construct these annotations, we develop a vision-language model pipeline that identifies subgoals from changes in objectives or interaction targets, anchors navigation goals to visually recognizable waypoints, and composes adjacent subgoals into task instructions. The resulting demonstrations cover navigation, reconnaissance, combat, interaction, and recovery, with explicit stage boundaries that connect task progression to fine-grained control. We assess annotation quality through model-based verification and human inspection. An audit of 500 trajectories yields a 95.8% pass rate across independent reviewer assessments. AceAction provides a data foundation for studying how agents execute, combine, and switch skills to follow complex instructions in dynamic game environments.
Community
I’m interested in the dataset mentioned in the paper but couldn’t find a download link. Is it publicly available, or are there plans to release it?
Thank you for your interest in our work! The dataset is currently undergoing our company’s internal legal review. We hope to release it once the necessary approvals are obtained and will share updates as they become available. Thank you for your patience and understanding.
Get this paper in your agent:
hf papers read 2611.04241 Don't have the latest CLI?
curl -LsSf https://hf.co/cli/install.sh | bash Models citing this paper 0
No model linking this paper
Datasets citing this paper 0
No dataset linking this paper
Spaces citing this paper 0
No Space linking this paper
Collections including this paper 0
No Collection including this paper