|
Download README.md from asher577/separate_normalization: direct link, hf CLI and curl.
- Browser
- Download file 1.06 kB
-
https://huggingface.co/asher577/separate_normalization/resolve/main/README.md
- Command line
-
hf download hf://asher577/separate_normalization/README.md
-
curl -L -o README.md https://huggingface.co/asher577/separate_normalization/resolve/main/README.md
1.06 kB
| # separate_normalization | |
| Weight-sparse transformer trained with the procedure from Gao et al. (2025). | |
| ## Model Details | |
| - **Layers**: 2 | |
| - **Model Dimension**: 3072 | |
| - **Context Length**: 512 | |
| - **Head Dimension**: 16 | |
| - **Vocabulary Size**: 4096 | |
| ## Sparsity | |
| - **Weight Sparsity**: True | |
| - **Target L0 Fraction**: 0.01 | |
| - **Activation Sparsity**: True | |
| ## Training | |
| - **Dataset**: jacobcd52/simplestories-tokenized | |
| - **Tokenizer**: SimpleStories/SimpleStories-1.25M | |
| - **Total Tokens**: 1,000,000,000 | |
| ## Training Run | |
| - **W&B Run**: [https://wandb.ai/asher577/nodesparse_adv_pretraining/runs/hhyouoq4](https://wandb.ai/asher577/nodesparse_adv_pretraining/runs/hhyouoq4) | |
| ## Usage | |
| ```python | |
| import torch | |
| from huggingface_hub import hf_hub_download | |
| # Download model | |
| model_path = hf_hub_download(repo_id="asher577/separate_normalization", filename="pytorch_model.bin") | |
| config_path = hf_hub_download(repo_id="asher577/separate_normalization", filename="config.json") | |
| # Load (requires the SparseGPT model class from this repo) | |
| state_dict = torch.load(model_path) | |
| ``` | |