Sat Grewal PRO

Your greedy question is the one place I already have data. On the Gemma 12B at 131K, temp 0 made it worse, not better: 12 of 25 samples looped deterministically until they died at the budget, and the same samples looped again on rerun. Vendor sampling (temp 1.0) halved the failures to 6 of 25 but did not eliminate them. Raw prediction files for both conditions are in the repo. On the 299B at IQ1_M it is one step darker: loops happen even with sampling, at short context, on trivial prompts. So the pattern across bit depths so far: at 4-bit you need greedy plus long context to trigger the loop; at 1.7-bit sampling cannot pull it out. That lines up with your flattened-logits read.

Adopting your loop-rate metric as of the next runs: the bridge will store full reasoning traces, and I will publish n-gram repeat rate next to each RULER rung, plus a bit-depth sweep across the Hy3 quant ladder as those rungs finish building. If loop rate climbs faster than accuracy falls, you called it.

That's a great idea actually 👍
Q3 and Q4 runners are being built right now anyway as part of the MTP requant wave, so once they exist I can compute imatrices from all three and publish the rank correlations across bit depths, whichever way they land.

On your question, today's hands-on testing already gave me half the answer: reasoning gives out first, and it gives out early. IQ1_M holds together for chat and short answers, but reasoning mode collapses into circular loops even on trivial prompts at short context. That behavior is documented on the card. Retrieval at depth is the untested half: the NIAH ladder for Hy3 is queued right after the MTP work, so whether retrieval outlasts reasoning toward the baked limit is exactly what it will show. Your point about extension rot before the nominal limit matches what I measured on other models too: my 6x and 8x YaRN experiments showed the factor itself taxing quality even while needles still passed. Numbers publish as the rungs finish.

You read it exactly right, and it's the part I'd poke at too. The bootstrap was physics, not preference: Hy3 f16 is 598GB and nothing I own can run it. I have two machines: a 5090 box (32GB VRAM, 128GB system RAM) and a MacBook Pro M3 Max with 128GB. I can only try what physically fits on these without a painfully slow dev experience. So the highest-fidelity runner available for computing the imatrix was my own Q2_K quant, loaded fully into the 5090 box's 128GB system RAM and running CPU-side (the card's 32GB can't hold it). A full-precision imatrix comparison doesn't exist yet, and I won't pretend otherwise. What does exist as sanity: all three quants passed coherence gates, and IQ2_M got a full day of hands-on today (reasoning converges at 2.5 bits, tool calls work, field notes are on the card including where IQ1_M breaks). imatrix v2 from a higher-fidelity runner is already on my list, and the low-bit rungs get requantized when it lands.

On 256K: correct, there are zero needle certifications on Hy3 yet, at any depth. That's why the card says EXPERIMENTAL in red and "1M baked but unverified." The NIAH ladder (262K first, then up) is queued right after the MTP work finishing now, and the numbers get published pass or fail, same as the gemma cards where I printed the misses too.

By the way, thanks to your earlier replies I went and adopted NVIDIA's RULER as the next test tier. First results are live on the Gemma4-12B card: five tasks at 131K, scored by NVIDIA's own evaluate.py, including a finding I didn't expect (thinking mode halves the score, and I published that too, zeros and all). Your kind of scrutiny is what pushed the testing up a level, so genuinely, thanks.

Since you're following this closely: MTP validated today on a llama.cpp fork port, 85.8% draft acceptance. Quants with the MTP head baked in are built and going through my own testing before I upload anything.

by then hopefully I'll have MTP also baked in. Hopefully 🤞
If happens, I'll push MTP quants. But my resources are not VRAM rich so that's why it takes this long in building/testing stuff.

Good push! But small correction, it is not single-needle. My harness plants 10 needles at 5-95% depths and the model has to return ALL 10 in one single response. The niah_test.py and raw transcripts are right there in the repo, you can check the results.jsonl. So it's 10 parallel retrievals held together, not recency + lookup.

But yes, you are right on the deeper part. My 10 needles are independent facts, nothing forces the model to combine fact A with fact B. So it tests retrieval, not composition. I kinda like this idea, so I am adding a hop tier to the harness: chained needles (needle 1 has the key that unlocks needle 2), and variable tracking (one value changes few times across the context, ask the final state). Will run it at 500K and 1M on the same models and publish the raw results either way, misses included, like everything else in these repos.