Right Screen, Wrong Transition: World Models as Verifiers for GUI Agents
Abstract
A decoder-free, action-conditioned world model predicts next-screen representations to verify GUI transitions and detect hijacked screens with high accuracy and low latency.
A login screen that appears after a tap on Sign in is expected; the same screen after a tap on View order is an attack. For GUI agents, safety is therefore a property of the transition rather than of the screen, and a monitor that inspects only screens can be defeated by reusing a legitimate one. Judging a transition requires an expectation of what should have followed the action. Existing GUI world models provide one, but they output it as text, code, or images, so checking it against the observed screen requires a second model to judge the two. We argue that a world model meant for verification should instead predict in the space in which observations are encoded, and present LGWM, a decoder-free, action-conditioned world model that predicts the representation of the next screen directly, trained without semantic annotation on 1.85M real GUI transitions. Verification reduces to a vector comparison, and the same signal reveals whether a mismatch is harmful. We evaluate on RSWT-BENCH, a diagnostic where each credential screen appears under both a legitimate and a hijacked transition, so detectors that see only the screen are at chance by construction. The training-free score reaches 0.987 AUC at 17 ms per decision, on par with the strongest closed-source VLMs and about ten AUC points above generative GUI world models at over three orders of magnitude lower latency. The residual direction reaches 0.953 AUC at separating harmful from benign violations, where prompted VLMs are near chance. Further analyses show that the prediction is a usable future state rather than an anomaly score. World models have mostly served as simulators or planners; our results point to a third role, verification, for which predicting in representation space is the natural design.
Community
Right Screen, Wrong Transition: World Models as Verifiers for GUI Agents
This is an automated message from the Librarian Bot. I found the following papers similar to this paper.
The following papers were recommended by the Semantic Scholar API
- GUI-CC: Benchmarking Contextual Consistency of GUI World Models as Agent Environments (2026)
- Decoupling What from Where: How Should a Small GUI Grounding Model Receive the Action Type? (2026)
- One Rollout Is All You Get: Fully Test-Time Adaptation for GUI Agents (2026)
- VeriCUA: Learning Proactive Verification for Reliable GUI Agent Rewards (2026)
- Beyond Next-Screen Prediction: The Decision Value of GUI World Models (2026)
- Future Anchored Verification and Online Recovery for World Action Models (2026)
- Proxy Confidence: Auditing Black-Box LLM Agents with a Surrogate's Log-Probabilities (2026)
Please give a thumbs up to this comment if you found it helpful!
If you want recommendations for any Paper on Hugging Face checkout this Space
You can directly ask Librarian Bot for paper recommendations by tagging it in a comment: @librarian-bot recommend
Get this paper in your agent:
hf papers read 2610.11942 Don't have the latest CLI?
curl -LsSf https://hf.co/cli/install.sh | bash Models citing this paper 1
Datasets citing this paper 1
jiamingzz/lgwm-data
Spaces citing this paper 0
No Space linking this paper
Collections including this paper 0
No Collection including this paper