Post
32
The rotary test cases are in and a solution was found.
https://huggingface.co/blog/AbstractPowered/rotary-addressed-selection
I've coined the term Rotary Addressed Selection, at least until I discover if someone else has made it first.
This rotary can handle an absurd length of context window, though the technical documentation is quite complex. Targeting context window after 1 million was how the addressing corruption was solved.
Yes this is all compacted into a 9m param model. A universe in an atom.
The rotary system involves including a full address for every 4096 context window lookup state. This allows the model to differentiate between a multitude of modular model extensions.
How does that work? Well if there's nothing at the address, then there's no return. Addresses are essentially selectors promoting the cooperative state and it does have some limitations, but the full global rotary supports DIRECT dynamic extension to the model at runtime.
Need only a simple classifier? The selectors know which experts classify and which state they exist in.
Need a full VLM? Load up the experts.
You get the idea.
Query with the address, get the correct addressing. That's how it works. The model learns to handle everything internally, so there is no need for a large structural addressing screening beforehand.
With this development, full mixture of experts with uncertain (absurdly long) context depth can have a consistent context window guarantee. There is bound to be some direct breakdowns and numerics that can't simply reach beyond certain points likely based on hardware or rounding differences, but the system essentially allows full MOE where experts are selectable and the selected experts are a guaranteed addressing.
The upcoming experimentation will fill in the gaps for the test cases, confirm or refute clauses that are established on the rotary, and then training a functional Alhena at the size of the mini-beatrix scale with some tweaks for the changes.
https://huggingface.co/blog/AbstractPowered/rotary-addressed-selection
I've coined the term Rotary Addressed Selection, at least until I discover if someone else has made it first.
This rotary can handle an absurd length of context window, though the technical documentation is quite complex. Targeting context window after 1 million was how the addressing corruption was solved.
Yes this is all compacted into a 9m param model. A universe in an atom.
The rotary system involves including a full address for every 4096 context window lookup state. This allows the model to differentiate between a multitude of modular model extensions.
How does that work? Well if there's nothing at the address, then there's no return. Addresses are essentially selectors promoting the cooperative state and it does have some limitations, but the full global rotary supports DIRECT dynamic extension to the model at runtime.
Need only a simple classifier? The selectors know which experts classify and which state they exist in.
Need a full VLM? Load up the experts.
You get the idea.
Query with the address, get the correct addressing. That's how it works. The model learns to handle everything internally, so there is no need for a large structural addressing screening beforehand.
With this development, full mixture of experts with uncertain (absurdly long) context depth can have a consistent context window guarantee. There is bound to be some direct breakdowns and numerics that can't simply reach beyond certain points likely based on hardware or rounding differences, but the system essentially allows full MOE where experts are selectable and the selected experts are a guaranteed addressing.
The upcoming experimentation will fill in the gaps for the test cases, confirm or refute clauses that are established on the rotary, and then training a functional Alhena at the size of the mini-beatrix scale with some tweaks for the changes.