README.md

---
license: apache-2.0
tags:
  - 3d-occupancy-prediction
  - autonomous-driving
  - self-supervised
  - nuscenes
  - arxiv:2511.17221
library_name: pytorch
---

# QueryOcc: Query-based Self-Supervision for 3D Semantic Occupancy

Weights for [QueryOcc](https://arxiv.org/abs/2511.17221) (Lilja, Lan, Fu, Hammarstrand),
**CVPR 2026**.

[Project page](https://research.zenseact.com/publications/queryocc/) ·
[Paper (arXiv)](https://arxiv.org/abs/2511.17221) ·
[Code](https://github.com/zenseact/queryocc)

QueryOcc learns continuous 3D semantic occupancy from multi-view camera images by supervising
directly in 4D space-time: positive and negative queries are sampled along rays from point
clouds observed in adjacent frames — no rendering losses, no voxelized lidar aggregation. No
human annotation is used at any stage. Two arms are published here, differing only in which
point clouds provide the supervision: the **camera-only** arm (pseudo point clouds from a
vision foundation model — the paper's headline model) and the **lidar-supervised** arm (real
lidar sweeps with per-point pseudo-semantics).

## Files

| File | Preset | Supervision | Sem. RayIoU | Dyn. RayIoU | Occ. RayIoU | Sem. IoU | Occ. IoU |
|---|---|---|---|---|---|---|---|
| `queryocc-nusc.pth` | `queryocc-nusc` | camera-only (pseudo) | 24.25 | 22.45 | 45.98 | 21.51 | 55.06 |
| `queryocc-lidar-nusc.pth` | `queryocc-lidar-nusc` | real lidar | 23.2 | 19.5 | 48.8 | 20.4 | 56.9 |

Occ3D-nuScenes validation split, self-supervised protocol. 112.1 M parameters, 449 MB each.

```
sha256  queryocc-nusc.pth        4a54ffb927fff5cb94086a14dfc02caaa99511ab2a97170fd6a2b1ffd0601604
sha256  queryocc-lidar-nusc.pth  561112071f8c1417b466798353dadaf5c54175b2107d25f8434839145c9d9dcc
```

Lidar supervision trades semantic accuracy for better occupancy, which is what its accurate
depth would predict.

### On the camera-only numbers

The paper reports 23.6 / 21.7 / 45.2 / 21.3 / 55.0 for `queryocc-nusc`. This checkpoint is
above the paper on all five, from a post-publication learning-rate sweep rather than any change
to the method: the peak LR is 7.5e-5 instead of the paper's 5e-5, and nothing else differs.
Architecture, input resolution (256×704), supervision heads and step budget are unchanged, so
the comparison to the paper's baselines is unaffected. The repository's `queryocc-nusc` preset
now defaults to 7.5e-5, and reproduces this checkpoint; pass `--model.max_lr=5e-5` for the
exact paper configuration.

Two caveats worth stating plainly. Occ3D-nuScenes has no held-out test split — these numbers
are the validation split, which is what the benchmark and the paper report on, and which the LR
sweep also selected on. And the metric carries real seed noise: three seeds of this
configuration scored 23.91 / 24.24 / 24.25 semantic RayIoU (mean 24.13). The published file is
the best of those three, so **24.13 is the honest expectation when retraining, and 24.25 is
what this particular file scores.** Every seed beat the paper's 23.6.

## Usage

```bash
git clone https://github.com/LiljaAdam/queryocc && cd queryocc
./setup_uv_env.sh && source .venv/bin/activate

python queryocc/train.py queryocc-nusc --no-train --test \
    --load_ckpt_from=hf://QueryOcc/queryocc
```

The `hf://<owner>/<repo>` spec resolves `<preset>.pth` inside the repo, so selecting
`queryocc-lidar-nusc` instead fetches the other file from the same URL. The checkpoint is
downloaded and cached automatically, and records the preset it belongs to — loading it into a
different preset is an error rather than a silent partial load. To use a manually downloaded
file, pass its path instead.

Evaluation additionally requires nuScenes, the Occ3D-nuScenes labels and a CUDA toolkit (the
RayIoU raycaster is JIT-compiled on first use); the repository README covers the setup. Nothing
else is needed — in particular, evaluating these weights does **not** require the gated DINOv3
backbone checkpoint that training uses.

Add `--allow_visualization --visualizer.f_viz_occ_3d_bev --visualizer.f_viz_bev_features` to log
BEV predictions and a PCA projection of the BEV features to Weights & Biases.

## Loading the weights directly

The file is a plain `torch.save` payload containing only tensors and primitives, so it needs no
trust in the publisher:

```python
import torch

payload = torch.load("queryocc-nusc.pth", map_location="cpu", weights_only=True)
payload["preset"]  # 'queryocc-nusc'
payload["metrics"]  # the numbers in the table above
payload["state_dict"]  # 'net.'-prefixed network weights
```

## Training data and intended use

Trained on nuScenes trainval: multi-view camera images, supervised either by pseudo point
clouds derived from monocular metric depth and vision-foundation-model features (camera-only
arm) or by lidar sweeps carrying offline per-point pseudo-semantics (lidar arm). Research
artifact: a demonstration of self-supervised occupancy learning, not a validated driving
component. Its outputs reflect the geometry, sensor suite and geographies of nuScenes (Boston
and Singapore) and should not be relied on for safety-critical decisions. Use is subject to the
[nuScenes terms](https://www.nuscenes.org/terms-of-use) as well as the license below.

## License

Apache 2.0, matching the code.

## Citation

```bibtex
@inproceedings{lilja2026queryocc,
  title={QueryOcc: Query-based Self-Supervision for 3D Semantic Occupancy},
  author={Lilja, Adam and Lan, Ji and Fu, Junsheng and Hammarstrand, Lars},
  booktitle={Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)},
  pages={21399--21408},
  year={2026}
}
```