Papers
arxiv:2611.04241

AceAction-5M: Hierarchical Goals for Generalist Action Learning Across 3D Game Worlds

Published on Nov 3
Authors:
,
,
,
,
,
,

Abstract

AceAction provides multi-game human gameplay demonstrations annotated with task instructions, ordered subgoals, local skill instructions, and temporally aligned visual-action data to support learning skill composition and subgoal transitions in dynamic 3D environments.

Video games provide a practical setting for collecting vision-language-action (VLA) data, whose acquisition in physical environments is costly. Existing datasets, however, remain limited by restricted game coverage, sparse language supervision, or temporal hierarchies that do not necessarily reflect changes in task objectives. We introduce AceAction, a dataset of human gameplay across multiple 3D games for learning skill composition and subgoal transitions. AceAction organizes demonstrations around complete tasks, linking overall instructions, ordered subgoals, and local skill instructions to temporally aligned visual observations and keyboard-and-mouse actions. To construct these annotations, we develop a vision-language model pipeline that identifies subgoals from changes in objectives or interaction targets, anchors navigation goals to visually recognizable waypoints, and composes adjacent subgoals into task instructions. The resulting demonstrations cover navigation, reconnaissance, combat, interaction, and recovery, with explicit stage boundaries that connect task progression to fine-grained control. We assess annotation quality through model-based verification and human inspection. An audit of 500 trajectories yields a 95.8% pass rate across independent reviewer assessments. AceAction provides a data foundation for studying how agents execute, combine, and switch skills to follow complex instructions in dynamic game environments.

Community

I’m interested in the dataset mentioned in the paper but couldn’t find a download link. Is it publicly available, or are there plans to release it?

·
Paper author

Thank you for your interest in our work! The dataset is currently undergoing our company’s internal legal review. We hope to release it once the necessary approvals are obtained and will share updates as they become available. Thank you for your patience and understanding.

Sign up or log in to comment

Get this paper in your agent:

hf papers read 2611.04241
Don't have the latest CLI?
curl -LsSf https://hf.co/cli/install.sh | bash

Models citing this paper 0

No model linking this paper

Cite arxiv.org/abs/2611.04241 in a model README.md to link it from this page.

Datasets citing this paper 0

No dataset linking this paper

Cite arxiv.org/abs/2611.04241 in a dataset README.md to link it from this page.

Spaces citing this paper 0

No Space linking this paper

Cite arxiv.org/abs/2611.04241 in a Space README.md to link it from this page.

Collections including this paper 0

No Collection including this paper

Add this paper to a collection to link it from this page.