|
Download README.md from asher577/separate_normalization: direct link, hf CLI and curl.
- Browser
- Download file 1.06 kB
-
https://huggingface.co/asher577/separate_normalization/resolve/main/README.md
- Command line
-
hf download hf://asher577/separate_normalization/README.md
-
curl -L -o README.md https://huggingface.co/asher577/separate_normalization/resolve/main/README.md
1.06 kB
separate_normalization
Weight-sparse transformer trained with the procedure from Gao et al. (2025).
Model Details
- Layers: 2
- Model Dimension: 3072
- Context Length: 512
- Head Dimension: 16
- Vocabulary Size: 4096
Sparsity
- Weight Sparsity: True
- Target L0 Fraction: 0.01
- Activation Sparsity: True
Training
- Dataset: jacobcd52/simplestories-tokenized
- Tokenizer: SimpleStories/SimpleStories-1.25M
- Total Tokens: 1,000,000,000
Training Run
Usage
import torch
from huggingface_hub import hf_hub_download
# Download model
model_path = hf_hub_download(repo_id="asher577/separate_normalization", filename="pytorch_model.bin")
config_path = hf_hub_download(repo_id="asher577/separate_normalization", filename="config.json")
# Load (requires the SparseGPT model class from this repo)
state_dict = torch.load(model_path)