SPAC-4B

Sparse Proximity-Aware Compression for 3D Understanding and Generation

Code & Demo · Training Data

One language model, two directions: describe a 3D mesh in words, or generate a mesh from a text description. SPAC combines a sparse SDF VQ-VAE, geometry-aware 3D-BPE, and a Qwen3-4B backbone with explicit spatial positional encoding.

Why sparse 3D tokens?

• Compute near the surface. Encode a narrow SDF band instead of an entire dense voxel volume. Under the paper's fixed voxel-band assumptions, active inputs grow approximately with surface area, O(R²).
• Compress in 3D. Merge recurring face-adjacent patterns using token pairs and their relative offsets. 3D-BPE is reversible at the discrete VQ-token level.
• Keep coordinates explicit. Serialize geometry with Morton-coordinate tokens and Spatial-PE so the language model can consume and predict 3D structure.

From text to meshes

Qualitative examples from the manuscript. SPAC predicts SDF geometry; colored appearances in the paper are visualization/postprocessing, while the standalone demo exports an untextured mesh.

Mesh understanding examples from the manuscript

Paper highlights

At an equal merge budget, 3D-BPE reduces the leaf-token sequence by approximately 24.6%:

Tokenizer	Remaining sequence length ↓	Bigram entropy ↓
3D-BPE	0.754	0.972
1D Morton BPE	0.803	1.092
1D Hilbert BPE	0.820	1.220
1D Random BPE	0.998	2.167

See the merge-budget sweep for additional compression and dependency measurements.

At R = 256, the manuscript reports 137.98 GFLOPs and 85.04 ms for the SPAC encoder, versus 41.38 TFLOPs and 4227.84 ms for the dense ShapeLLM encoder, under its reported protocol.

Text-to-3D results reported in the manuscript

Method	CLIP ↑	FD ↓	KID ×100 ↓
SAR3D	23.9	27.2	0.28
TRELLIS	30.8	18.3	0.10
ShapeLLM-Omni	26.7	25.9	0.25
SPAC	24.4	23.7	0.22

These figures and tables describe the paper's experimental models and protocols. They are not fresh benchmark measurements of the checkpoint distributed on this page. In particular, the latest Toy4k training-overlap replay is a different experiment.

Run SPAC-4B

Requires Linux x86_64, CUDA Toolkit 12.4 (nvcc), a C++ compiler, Google SparseHash headers, and a compatible NVIDIA driver. An A100 40 GB is recommended. Both the LLM and VQ-VAE run in FP32.

# Ubuntu build prerequisites
sudo apt-get install -y git build-essential libsparsehash-dev
export CUDA_HOME=/usr/local/cuda-12.4  # use your CUDA 12.4 installation
export PATH="$CUDA_HOME/bin:$PATH"

git clone https://github.com/yangjj007/SPAC-main.git
cd SPAC-main
bash install.sh --dependencies-only
.venv/bin/python download_weights.py
CUDA_VISIBLE_DEVICES=0 .venv/bin/python check_model_compatibility.py --check-vae

TorchSparse is built from source following Direct3D-S2. The installer runs the upstream clone-and-install recipe automatically, pinned to TorchSparse 2.1.0; SPAC retains its tested PyTorch 2.6.0 / torchvision 0.21.0 / CUDA 12.4 stack.

git clone https://github.com/mit-han-lab/torchsparse.git .cache/torchsparse
git -C .cache/torchsparse checkout 385f5ce8718fcae93540511b7f5832f4e71fd835
.venv/bin/python -m pip install --no-build-isolation .cache/torchsparse

The downloader pins a compatible release and installs/verifies the LLM, VQ-VAE, BPE merge table, and Spatial-PE manifest at their explicit runtime paths. It disables proxy use. To download through a mirror, append --endpoint https://hf-mirror.com. Repository authentication may be required when cloning the code.

# Text → untextured mesh, token sequence, and runtime settings
.venv/bin/python infer.py generate "A wooden barrel with metal hoops." --output outputs/barrel

# Mesh → caption, using live mesh encoding
.venv/bin/python infer.py understand examples/barrel/reference.glb --encode-mesh

# Interactive interface
CUDA_VISIBLE_DEVICES=0 bash run.sh --host 127.0.0.1 --port 7860

The interface opens at http://localhost:7860. Packaged examples can reuse their saved input tokens; --encode-mesh runs the complete mesh-to-SDF-to-token path.

Runtime file mapping

Hugging Face path	Local runtime path
Four pytorch_model-*.bin shards, index, config and tokenizer	models/llm/
merge_table.json	models/merge_table.json
spatial_pe_manifest.json	models/spatial_pe_manifest.json
vqvae/model.pt, vqvae/config.json	models/vqvae/

The downloader records these mappings and SHA-256 checksums in models/asset_manifest.json. The VQ-VAE filename change preserves the checkpoint bytes.

Inference defaults

Setting	Released demo
Precision / attention	FP32 / SDPA; TF32 and AMP disabled
Spatial-PE / RoPE theta	0.08 / 1,000,000
Generation	Greedy; temperature 0, top-k 1, top-p 1; up to 8192 new tokens
Understanding	Greedy; up to 256 new tokens
Geometry	SDF resolution 256; explicit coordinate grid 32³

The release config declares FP32, RoPE theta 1,000,000, and independent input/output embeddings. The loader also handles the original export's stale tied-embedding flag without overwriting the learned lm_head. Geometry inference requires the SPAC wrapper and matching spatial manifest / merge table; standard text-only AutoModel.generate() does not add Spatial-PE. An output that reaches the token limit can be incomplete; the demo records ended_normally in result.json.

Checkpoint provenance

SPAC-4B uses the LLM checkpoint originally released as Jack04810/SPAC-4B-toy4k, revision de08d429d2910e784b7ca0a22731246e26649c06 (checkpoint-20352). It is a Toy4k-continuation checkpoint exported with FP32 weights and a 200,832-token vocabulary. Renaming the release does not change its training history. The repository also provides the matching VQ-VAE under vqvae/.

The method is described in “From Dense to Sparse: Morton-Ordered Sparse Mapping for Efficient 3D Mesh Generation.” The model generates geometry without native textures or materials. License metadata is retained from the source checkpoint.