Muir: Linear Attention with Sensitivity-Allocated Gating and Matched-Filter Readout

Great question!! Yes, it actually helps even more in (extreme) long contexts. Because the key energy concentration (~6-7 128 channels) is a per-token property of SiLU+L2-norm, even when the hidden state is evolved over millions of tokens, each single-step erasure only acts on ~6-7 channels. In fact, using a low-rank for erase gate is much more crucial for long-context modeling because it suppresses non-normal oblique residual (Rem. 3.3 in the paper) that would otherwise inflate across millions of steps.

Suppressing this non-normal residual (+ wider FFN) is also one of the reasons why Muir variant w/out the read gate also outperforms GDN-2. Hope this helps!! :)

We haven't tested on multilingual corpora yet, but your question/setting hits on two really neat properties of Muir.

1- Because key sparsity is a geometric property of SiLU+L2-norm (Prop. 3.1), even if German and Danish paragraphs activated completely different subsets of channels, each token still concentrates its energy onto ~6-7 (of 128) channels, and the token-dependent low-rank projection dynamically modulates erasure across whichever channels fire.

2- A shift in tokenization density can be seen as a shift in token lag over which a semantic dependency has to survive (= a language with higher tokenization density has a longer lag). This is exactly where the token-dependent read gate would really help: as token density increases and stretches effective lags, the read gate can shift weight toward slow-decaying rows on the fly!!

I think a multilingual/multi-domain mix is an interesting (and natural) testbed for Muir.

Yes, the write/erase asymmetry matters even more at extremely long contexts. From a "you don't need this much" standpoint, because SiLU+L2-norm concentrates each key onto ~6-7 (of 128) channels regardless of sequence length, a single-step erasure doesn't need higher rank. But it gets better!! Over millions of tokens, a dense erase gate injects jitter that inflates the spectral norm of the transition matrix (Rem. 3.3) and compounds over time; on the other hand, a low-rank erase gate projects the noise out. This is what primarily helps Muir (even without the read gate) extrapolate well beyond GDN-2.

Aside: In thinking about your comment, I looked up if write/erase asymmetry also applies to us humans. Funnily enough, any given memory cue apparently only lights up about 2-5% of neurons in the hippocampus (just like our sparse keys). Recording a new memory requires detailed rewiring across thousands of specific connections, while erasing/updating an old memory only takes a simple chemical switch acting on the few neurons that just fired. (I'm not a neuroscientist in any way, so I'm not sure if this is entirely accurate or not :))

Hope this helps!!