Trace-Anything — transformers format (original code)

Unofficial transformers port of Trace Anything (depth-anything/trace-anything, trace_anything.pt) that runs the original code. Loads with trust_remote_code. model.safetensors holds the checkpoint's tensors under their original names (strict load, every tensor checked equal).

Not an official release. Trace Anything was created by Xinhang Liu, Yuxi Xiao, Donny Y. Chen, Jiashi Feng, Yu-Wing Tai, Chi-Keung Tang and Bingyi Kang — https://github.com/ByteDance-Seed/TraceAnything, paper "Trace Anything: Representing Any Video in 4D via Trajectory Fields" (arXiv 2510.13802).

What it is

A video in, a trajectory field out: for every pixel of every frame, the control points of a B-spline 3D trajectory over time (and their confidence), in one forward pass. TraceAnythingModel holds the official TraceAnything's submodules (encoder: CroCo ViT-L, decoder: 24-block trace decoder, ds_head_time / ds_head_track: DPT heads) and runs the official TraceAnything.forward; TraceAnythingOutput stacks its per-view results (time, ctrl_pts3d, ctrl_conf, track_pts3d, track_conf) and keeps the original list as predictions. Code vendored from ByteDance-Seed/TraceAnything @ 54677b5e into ta_layers.py / ta_heads.py / ta_model.py (imports only). TraceAnythingProcessor (processing_trace_anything.py, processor_config.json) wraps TraceAnythingImageProcessor, scripts/infer.py's _load_images.

Local head left out. configs/eval.yaml enables a second ("local") trajectory head, but the released checkpoint holds no weights for it: infer.py loads with strict=False, so that head runs with random initial weights and its *_local outputs carry no trained signal. This package sets head_args.with_local_head: false; all other outputs are those of the original.

Verified against the official code (same process, V100; infer.py's model build and _load_images): preprocessing and every output (time, control points, confidences, evaluated tracks; also as a tuple) are bit-identical on the repo's example scenes (corgi 2 frames, cabinet 11, the first 8 / 16 frames of elephant and monkeys), with the original's pytorch_naive attention. flash_attention (eval.yaml) and pytorch_auto have no bf16 kernel on a V100, so they were not run; the vendored code is the same for all three.

Usage

import torch
from transformers import AutoModel, AutoProcessor

model = AutoModel.from_pretrained("Trace-Anything", trust_remote_code=True).eval().cuda()
processor = AutoProcessor.from_pretrained("Trace-Anything", trust_remote_code=True)
inputs = processor(images=frame_paths, return_tensors="pt").to("cuda")   # the frames of one video, in order
outputs = model(**inputs)    # outputs.ctrl_pts3d: (N, 10, H, W, 3); outputs.track_pts3d: (N, N, H, W, 3)

• Attention kernel: config.encoder_args["attn_implementation"] / config.decoder_args[...] — "flash_attention" (eval.yaml; needs compute capability 8.0+), "pytorch_auto" or "pytorch_naive" (the original's options). Change both on the config before from_pretrained(..., config=config) on older GPUs.
• One video per call. infer.py keeps every (N // 39)-th frame of scenes with more than 40 frames; do that before the processor if needed.
• return_dict=False returns a tuple.

Runtime packages beyond torch / transformers: einops, opencv-python, torchvision (as the original; the original's optional CUDA RoPE extension is not used).

Licensing

• Weights: CC-BY-NC-4.0 (non-commercial) — LICENSE, with the upstream notice LICENSE_MODEL.txt.
• Code (vendored files and the port): Apache-2.0 — LICENSE.code.
• Attribution, sources and the changes made here: NOTICE.

Citation

If you use this model, please cite the original work, as its authors ask:

@misc{liu2025traceanythingrepresentingvideo,
      title={Trace Anything: Representing Any Video in 4D via Trajectory Fields}, 
      author={Xinhang Liu and Yuxi Xiao and Donny Y. Chen and Jiashi Feng and Yu-Wing Tai and Chi-Keung Tang and Bingyi Kang},
      year={2025},
      eprint={2510.13802},
      archivePrefix={arXiv},
      primaryClass={cs.CV},
      url={https://arxiv.org/abs/2510.13802}, 
}