OpenNWM

A Generalist Navigation World Model
with Latent Action Pretraining

OpenSpace Lab

Learn navigation dynamics from action-free video. Adapt them to physical world.

News · Setup · Data · Models · Evaluation · Training · TODO List

News

• 2026-11-03 — The paper is available on arXiv.
• 2026-11-01 — Updated the paper figure and documentation. Code, models, and NavAnywhere data are available.

Environment Setup

Requires Linux, Conda, and an NVIDIA GPU compatible with CUDA 12.4. Run from the repository root; LAM and NWM use separate environments.

git clone https://github.com/OpenSpace-Lab/OpenNWM.git
cd OpenNWM
for module in lam nwm; do
  conda create -n "open-$module" python=3.10 -y
  conda run -n "open-$module" python -m pip install torch==2.6.0 torchvision==0.21.0 \
    --index-url https://download.pytorch.org/whl/cu124
  conda run -n "open-$module" python -m pip install -r "$module/requirements.txt"
done

conda run -n open-nwm python setup_paths.py \
  --data-root /data/navigation --work-root /data/opennwm --nav-profile 15 \
  --lam-python "$(conda run -n open-lam python -c 'import sys; print(sys.executable)')" \
  --nwm-python "$(conda run -n open-nwm python -c 'import sys; print(sys.executable)')"
source .env

Replace /data/navigation and /data/opennwm with your data and work directories. Run source .env in each new terminal. To reuse existing splits, add --split-root /path/to/splits to setup_paths.py.

Data Preparation

Download NavAnywhere from Hugging Face or ModelScope and extract it to $NWM_DATA_ROOT. The archives total 1.93 TB; extraction requires tar, zstd, and additional disk space.

Download and extract with ModelScope

"$NWM_PYTHON" -m pip install modelscope
conda run -n open-nwm ms-hub download LittleBoss/NavAnywhere --repo-type dataset \
  --local-dir "$OPENNWM_WORK_ROOT/downloads/NavAnywhere" --max-workers 8
(
  cd "$OPENNWM_WORK_ROOT/downloads/NavAnywhere" || exit 1
  mkdir -p "$NWM_DATA_ROOT"
  find NavAnywhere -path '*/packs/*.tar.zst' -print0 \
    | xargs -0 -r -I{} tar --use-compress-program=unzstd -xf '{}' -C "$NWM_DATA_ROOT"
  for first in LAM-Data-{recon,scand,sacson}-*.tar.zst.part-00000; do
    [ -f "$first" ] || continue
    cat "${first%.part-00000}.part-"* \
      | tar --use-compress-program=unzstd -xf - -C "$NWM_DATA_ROOT"
  done
)

The action-data archives retain their LAM-Data- filenames. Expected layout:

$NWM_DATA_ROOT/
├── recon/<trajectory>/{0.jpg,1.jpg,...,traj_data.pkl}
├── sacson/<trajectory>/{0.jpg,1.jpg,...,traj_data.pkl}
├── scand/<trajectory>/{0.jpg,1.jpg,...,traj_data.pkl}
├── tartan/<trajectory>/{0.jpg,1.jpg,...,traj_data.pkl}
└── NavAnywhere/<source>/<trajectory>/{0.jpg,1.jpg,...}

The release includes RECON, HuRoN (sacson), SCAND, and NavAnywhere. TartanDrive requires a separate download for world model post-training and default LAM evaluation; follow the NWM data instructions and place it in tartan/ (split name: tartan_drive). setup_paths.py prepares the fixed train/validation/test lists and NavAnywhere recipe.

Pretrained Models

Download from Hugging Face or ModelScope into $OPENNWM_WORK_ROOT/models/OpenNWM, including configs/.

Checkpoint	Use
lam-nav15-pixel-action-step60k.ckpt	Latent action extraction
opennwm-nav15-stage1-pretrain60k.pth.tar	World model pretraining checkpoint
opennwm-nav15-stage2-joint100k.pth.tar	Physical-action prediction and planning

Download and verify with ModelScope

"$NWM_PYTHON" -m pip install modelscope
conda run -n open-nwm ms-hub download LittleBoss/OpenNWM --repo-type model \
  --local-dir "$OPENNWM_WORK_ROOT/models/OpenNWM" --max-workers 4
(cd "$OPENNWM_WORK_ROOT/models/OpenNWM" && sha256sum -c SHA256SUMS)

NWM downloads SD-VAE on first use; set NWM_VAE_PATH in .env to use a local copy. DINOv3 is needed only for LAM variant 3.

Evaluation

After preparing test data, import the Stage 2 checkpoint and run prediction:

"$NWM_PYTHON" run.py nwm import-checkpoint \
  --checkpoint "$OPENNWM_WORK_ROOT/models/OpenNWM/opennwm-nav15-stage2-joint100k.pth.tar" \
  --config "$OPENNWM_WORK_ROOT/models/OpenNWM/configs/opennwm-nav15-stage2.yaml" \
  --run-dir "$OPENNWM_WORK_ROOT/runs/released-stage2"
"$NWM_PYTHON" run.py nwm infer \
  exp_dir="$OPENNWM_WORK_ROOT/runs/released-stage2" ckp=latest eval_type=time \
  prediction_dir="$OPENNWM_WORK_ROOT/eval/stage2/pred"

Metrics, rollouts, and planning: NWM evaluation. Latent action reconstruction: LAM evaluation.

Training

Follow LAM training → cache preparation → world model pretraining and adaptation. Use the NavAnywhere 15-source profile and matching LAM checkpoints for caches. For the released model, set LAM_PIXEL_ACTION_CHECKPOINT in .env to $OPENNWM_WORK_ROOT/models/OpenNWM/lam-nav15-pixel-action-step60k.ckpt.

TODO List

• Release code, data, and pretrained models.
• Publish the paper, project page, and demos.
• Provide training and evaluation guides.
• Support closed-loop navigation on Unitree Go2 (deployment plan).

Acknowledgements & Citation

We thank NWM, CompACT, DreamDojo, and the dataset authors.

If you use OpenNWM, please cite:

@misc{nie2026opennwmgeneralistnavigationworld,
      title={OpenNWM: A Generalist Navigation World Model with Latent Action Pretraining},
      author={Dujun Nie and Ruiqi Song and Zixuan Wang and Hanqing Wang and Yifan Han and Baiyong Ding and Wenzhao Zheng and Hangbin Wu and Long Chen},
      year={2026},
      eprint={2611.01844},
      archivePrefix={arXiv},
      primaryClass={cs.RO},
      url={https://arxiv.org/abs/2611.01844},
}

NWM-derived components retain CC BY-NC 4.0; DreamDojo LAM components retain Apache 2.0. Datasets and external models retain their own licenses.

Model release details

Checkpoints

File	Use	Bytes
opennwm-nav15-stage2-joint100k.pth.tar	Stage 2 real-action navigation; 3,000 adapter warmup + 100,000 joint steps	3,137,820,686
opennwm-nav15-stage1-pretrain60k.pth.tar	Stage 1 latent-action pretraining; 60,000 steps	3,137,820,971
lam-nav15-pixel-action-step60k.ckpt	Nav15 PixelActionLAM; 60,000 steps; 32-dimensional latent actions	8,518,856,077

Use Stage 2 for real-action prediction and planning. Stage 1 uses latent actions and has not been adapted for direct real-action navigation. The LAM maps image pairs to 32-dimensional latent actions. These are the final Nav15 checkpoints, distinct from the older Nav1 weights in LittleBoss/LAM-weights.

Files preserve their original checkpoint format and bytes, including training state and historical configuration metadata. No optimizer tensors were removed and no weight conversion was performed. Use the separate portable YAML configs below instead of relying on historical machine paths embedded in checkpoints. All payload files are covered by SHA256SUMS; manifest.json records checkpoint roles, byte sizes and hashes.

Model and inference settings

The world models use CDiT-B/2 with four 224×224 context frames and stabilityai/sd-vae-ft-ema (latent scaling 0.18215). The VAE is an external dependency and is downloaded separately. Stage 2 uses the checkpoint's ema state, three real action dimensions, and 250 DDPM sampling steps. The real-action normalization bounds are [-2.5, -4, -π] to [5, 4, π]. LAM images use a 4:3 center crop followed by resize to 240×320; use the repository preprocessing code.

Portable configuration notes

NWM_VAE_PATH can point to a local VAE or use the default public identifier. Dataset evaluation also requires NWM_DATA_ROOT and NWM_SPLIT_ROOT; use the original split lists for historical score reproduction. The portable configs are for inference and do not claim exact training resumption. Refer to the repository NWM README for prediction, evaluation and planning, and the LAM README for using lam-nav15-pixel-action-step60k.ckpt with LAM evaluation or latent extraction. Stage 1's config additionally uses NWM_NAVANYWHERE_ROOT, NWM_NAVANYWHERE_RECIPE and NWM_LATENT_PROXY_ROOT.

Provenance and licenses

NWM code derives from Meta NWM and CompACT; its upstream CC BY-NC 4.0 license is preserved in licenses/NWM-CC-BY-NC-4.0.txt. DreamDojo LAM code's Apache 2.0 license is preserved in licenses/DreamDojo-Apache-2.0.txt. The repository code provenance pins DreamDojo 8f3eeb7, CompACT 8367a5e, and Meta NWM 3f6cd8e. This release retains both upstream notices and does not relicense upstream material. External VAE and data sources retain their respective terms.

The weights support research on visual navigation and future observation prediction. They are not a complete robot controller; predictions can drift outside the training distribution. No new benchmark scores or deployment claims are introduced by this release.