AutoCompress: Automated Hardware-Guided Model Compression
Abstract
AutoCompress is a hardware-guided compression framework that uses differentiable architecture search to optimize global and per-layer structures under serving constraints, improving accuracy over Minitron and compressing a large model while retaining most benchmark performance.
LLM pruning must preserve quality while meeting hardware- and workload-specific constraints on latency, throughput, and memory. Existing pruning workflows often target parameter counts, requiring practitioners to manually select model sizes that meet latency, throughput, and GPU memory limits. However, parameter count alone does not predict these metrics, which depend on the model architecture, workload, hardware, and serving engine. We introduce AutoCompress, a hardware-guided compression framework that directly incorporates such constraints into differentiable architecture search. AutoCompress jointly optimizes global and per-layer structure to produce heterogeneous architectures, guided by serving-engine cost models validated against end-to-end latency, throughput, and memory measurements. We evaluate AutoCompress on long-context local coding, batched datacenter serving, and latency-sensitive in-game interaction. Compared to the state-of-the-art Minitron pruning framework, AutoCompress improves average accuracy across eight benchmarks by 6.57, 2.35, and 0.57 percentage points on RTX 5070, RTX 5080, and H100, respectively. AutoCompress also compresses Nemotron 3.5 Super from 120B to 74B parameters for single-B200 BF16 serving at 128K context, retaining 95% of its mean benchmark score.
Community
Introducing AutoCompress: Automated Hardware-Guided Model Compression.
Give it a model and a latency and memory constraint for your deployment, and AutoCompress automatically produces a compressed model that meets them. No manual architecture design.
How it works: AutoCompress builds cost models through efficient, accurate profiling of the model on the target GPU. A differentiable architecture search then automatically picks an architecture that fits the constraints. Finally, we prune the model to the chosen architecture and distill it to recover accuracy.
Results:
✅ We compressed models for three diverse real-world use cases: an on-device game NPC (RTX 5070), a long-context coding agent (RTX 5080), and batched datacenter serving (8×H100). AutoCompress improves average accuracy by +6.6 / +2.4 / +0.6 points over the best baseline available (Minitron), with 2–3.3× fewer architecture-search tokens.
✅ At scale, we compressed Nemotron 3.5 Super from 120B to 74B parameters while keeping about 95% of its mean benchmark score.
The search produces heterogeneous per-layer architectures guided by serving-engine cost models. For the RTX 5070 local coding case, which engine did the cost model target? Many people serve those models through llama.cpp with GGUF quantization, and per-layer widths that differ can break conversion or lose the fast kernels. Did you check whether the pruned models still convert and quantize cleanly there?
Get this paper in your agent:
hf papers read 2611.08290 Don't have the latest CLI?
curl -LsSf https://hf.co/cli/install.sh | bash Models citing this paper 0
No model linking this paper
Datasets citing this paper 0
No dataset linking this paper
Spaces citing this paper 0
No Space linking this paper
Collections including this paper 0
No Collection including this paper