Vladimir Beskorovainyi

I ran the experiment rather than guess: a true second seed of the v1.3 recipe. It uses seed 1337, the seed of seed-aug-s2, on a rebuilt v1.3 mixture (198 intent forms, checked before training). The reading rule was committed before the run (633a7bf), and the epoch was fixed in advance at the third of four. The rule: the recipe gap is established on a set if it exceeds both within-recipe spreads.

Scientific rubrics, median over the twenty tens: the gap is +0.078, against spreads of -0.039 for v1.3 and +0.070 for v1.1. It clears the larger one by 0.008 and holds in 11 of 20 tens. So the answer to your question is nearer +0.078 than +0.025, but only just above seed noise. At one seed the label forms are worth -0.007 there, so whatever separates the recipes on this set, it is not the forms.

Jargon is where the forms do the work: the gap is +0.182 and +0.159 at eighteen and five candidates, against spreads of at most 0.074, and at one seed the forms alone are worth +0.262 and +0.201. On news and emotions the gap is -0.022 and -0.039 and not established, and the v1.3 recipe moves by 0.086 between its own two seeds on emotions. The fixed epoch is also not the best epoch of the new run on those two sets, which fits your point that one unseen set should not pick the checkpoint.

Everything is in results/v13_seed2/ in release v1.3.11 (https://doi.org/10.5281/zenodo.24185900), and the preprint’s v4 reports it, together with your point that the earlier intervals priced rows and labels but not the seed. Thank you, this question produced the most useful run of the project.

Your per-class numbers reproduce: on the alphabetical ten, Литература is +0.408 against v1.1, v1.1 answers it 245 times against 99 gold, and without the four prefix rubrics v1.3 minus seed-base-s2 is +0.026, [-0.011, +0.063] over rows and [-0.011, +0.059] over labels.

The 28-class matrix has landed, so I ran exactly your question, and one step further. Over twenty random tens of all 28 rubrics (seeds 0 to 19), v1.3 minus seed-base-s2 after biases has a median of +0.065 (range +0.037 to +0.101). The row interval clears zero in 20 of 20, and the label interval also in 20 of 20. Over twenty random tens drawn only from the 24 rubrics without a prefix match, the median is +0.066 (range +0.030 to +0.110), with the row interval clearing zero in 20 of 20 and the label interval in 19 of 20.

So you were right that on the alphabetical ten the lead over the second seed sits on a few classes, and that ten turns out to be one of the weakest draws for this comparison. Over the draws, the separation does not depend on the prefix rubrics or on which ten is chosen. The usual caveat stands: the v1.1 recipe has two runs here, and v1.3 has no second seed at all.

On the main question, the tie-break does not drive the ranking result either: over the same twenty tens, Spearman with news has a median of -0.19 and with emotions -0.20, and none of the forty reaches 0.5.

All of this was decided after the reading rule and is labelled so. The matrix and the sensitivity script are in v1.3.9 (https://doi.org/10.5281/zenodo.24152275). The prefix check is results/analysis/matrix_prefix.py and results/unseen3/prefix.md on main.

Your six numbers reproduce on v1.3.8, and a third unseen set is now scored. I recorded the reading rule before downloading any rows (commit 4d23c2d, 09:17 today, on the ladder-third-set branch): a ranking “survives” a new set if Spearman with it, after biases, is at least 0.5. The new set is scientific rubrics from ru-SciBench (GRNTI), 990 rows, 5 candidates, no text overlap with training.

Neither ranking survives. After biases, Spearman of the scientific rubrics with news is -0.18 and with emotions -0.41; as published, -0.41 and -0.25. All three pairwise correlations are negative, and each set picks a different winner: v1.3-ep2of4 on news, v1.2 on emotions, seed-aug-s3 on scientific rubrics. Within each set the ranking before and after the bias fit agrees (0.35, 0.94, 0.89), so the correction moves levels, not the order across sets. At n = 10 this is a description, not a test. So no single unseen set should pick the checkpoint for the others.

One difference does hold there. On the scientific rubrics v1.3 is ahead of v1.1 after biases by +0.091, [+0.059, +0.125] over rows and [+0.010, +0.184] over labels, and ahead of the second v1.1 seed by +0.045, with both intervals above zero. It is the first unseen set where the release gap is larger than the seed spread.

Your questions also made me run the same checks on the probe vocabularies, and they found things before anyone else did. A single label, qa, carries most of several vocabulary differences. v1.1 almost never picks it, while v1.3 does. The abbreviation gap of -0.060 is 52 of its 58 questions on qa and goes to 0.000 after biases. On strict jargon the v1.1 recipe spans 0.550 to 0.648 across seeds, with the release at the bottom. Jargon is the one gain that survives the correction intact. This is in release v1.3.8 (https://doi.org/10.5281/zenodo.24125472), and the preprint will be corrected accordingly.

One caveat. The ten rubrics were chosen by a tie-break, because all 28 have the same test size. A full score matrix over all 28 is running now, so that twenty random sets of ten can be checked. I will post that here when it finishes.

I scored every ladder checkpoint on the emotions rows and ran the same bias fit on both datasets. The answer is no: epoch 2 does not lead there. It is 0.563 against 0.571 for the released epoch 3, a difference of -0.008 with both intervals covering zero ([-0.030, +0.014] over rows, [-0.110, +0.040] over labels). The best checkpoint with biases on emotions is v1.2 at 0.591.

So the stopping-rule effect is real on the headlines but does not carry to the second unseen dataset. The other run that asks the same question, an early against a late epoch of another seed, gives +0.030 on headlines and +0.003 on emotions, the same pattern.

Your first point holds on both datasets. After the biases, v1.3 against the second seed of the v1.1 recipe is +0.004 on headlines and -0.006 on emotions, all intervals covering zero, while that seed differs from v1.1 by -0.037 and +0.020. One run per recipe cannot separate the recipe from the seed.

The size of the correction also turns out to measure how hard a checkpoint leans: the bias gain on headlines runs from +0.021 for v1.1 to +0.139 for epoch 2.

The table, the logits for both datasets and the scoring script are in results/unseen/ladder_bias.md, results/unseen/ladder_emotions/ and results/analysis/score_ladder.py, archived as release v1.3.7 (https://doi.org/10.5281/zenodo.24096621). Thank you for pushing on this, it made the repository more complete.

Thank you, this is a sharp reading. I re-ran unseen_bias.out_of_fold on every file in ladder/ and get exactly your numbers, including +0.053 [0.032, 0.075] over rows and [0.030, 0.079] over labels for epoch 2 against the released epoch 3.

I agree with the first point. After the biases, the release gap on headlines is the same size as reseeding the v1.1 recipe, so it cannot be attributed to the recipe.

On the stopping rule, two caveats. Epoch 2 is the best of eleven scored checkpoints and was picked after seeing the scores, so its interval is optimistic, and the label-level interval rests on six labels. On the other side, the pattern is not a single cell: the epoch-3 run of another seed (0.681) and v1.2 (0.664) also sit above the released epoch.

I do not have epoch 2 logits on the emotions set yet. I will score every ladder checkpoint on the emotions rows, run the same bias fit on both datasets and post the table here with the scoring script in the repository. If epoch 2 leads on emotions as well, I will report the stopping rule as a finding.

Here are the numbers, from results/unseen/ladder.md in v1.3.3. Your figures reproduce exactly.

Per checkpoint: accuracy; margin AUC (95% interval of the difference from v1.1); median paired change on economy headlines; economy headlines that now rank politics above economy.

v1.1: 0.652; 0.928; 0.00; 39
v1.1 recipe, second seed: 0.601; 0.909 [-0.044, +0.005]; -0.79; 58
v1.2: 0.577; 0.929 [-0.024, +0.023]; -2.33; 76
v1.3 schedule without transliteration, seed 2: 0.600; 0.911 [-0.041, +0.005]; +0.10; 60
same, seed 2, epoch 3: 0.601; 0.921 [-0.028, +0.013]; -0.68; 62
same, seed 3: 0.556; 0.834 [-0.133, -0.059]; -3.97; 107
v1.3 epoch 1: 0.526; 0.921 [-0.025, +0.011]; -2.61; 39
v1.3 epoch 2: 0.554; 0.938 [-0.008, +0.029]; -2.65; 66
v1.3 epoch 3, released: 0.568; 0.895 [-0.061, -0.008]; -2.66; 84
v1.3 epoch 4: 0.551; 0.875 [-0.088, -0.023]; -3.40; 88

The two things you measured move at different steps. The shift of economy headlines towards politics comes with v1.2, which added phrase-form wording and no transliteration (-2.33), and the first epoch of v1.3 brings it to -2.61, after which it stays flat. The margin AUC holds much longer and falls only at the released third epoch and the fourth.

Transliteration is not the cause: the checkpoints trained without any transliterated forms give 0.911, 0.921 and 0.834, the worst of the ladder, and a second seed of the v1.1 recipe gives 0.909. So the loss of separation sits inside the seed spread, and the drift towards politics arrived with phrase-form wording. By margin AUC the best v1.3 checkpoint is the second epoch, which we did not ship because it was lower in accuracy.

Yes, there is a ladder: v1.2, every epoch checkpoint of v1.3, a second seed of the v1.1 recipe, and two runs of the v1.3 schedule on a v1.2-level mix without the transliterated forms. I reproduced your numbers (AUC 0.928 and 0.895, 39 against 84, medians -0.09 and -2.66) and will score all of these on the 1,000 headlines, publish the logits and the margin AUC per checkpoint in the repository, and post the table here.

I get the same: after the biases politics recall 0.698 to 0.616 and economy 0.735 to 0.657, the two together about -2.7 of the -3.4 points, and both releases at 0.771 on politics before the fit. Before any fit v1.3 also sends 71 economy headlines to politics against 35 for v1.1. So I agree, what is left is a boundary between two labels, and a per-option shift cannot fix it.

On your question: no. Neither training mix has a news-topic task or a politics or economy label. The nearest are two MASSIVE intents present in both releases, qa_currency and qa_stock, as "курс валют" and "котировки акций"; v1.3 adds transliterated and clipped forms of the intent keys. Nothing in the mix files economic news under politics, so the data do not explain why v1.3 draws this boundary worse. The per-row logits are in results/unseen if you want the margin between the two options on the 174 economy headlines.

The per-option biases are now in main: scripts/fit_temperature.py --biases fits them next to the temperature on the same labelled rows, and the serving API applies them from calibration.json. results/analysis/unseen_bias.py reproduces the numbers above. Thank you for the push.

One correction to my earlier note on emotions. After the biases v1.3 is ahead by 1.4 points, and the interval includes zero both over rows and over labels, so I would not call it ahead. The biases are now in the calibration script in the calib-bias branch, with a script that reproduces these numbers.

Thank you for running this on the released rows. I reproduced it: 318 политика picks against 166 gold, 41 of the 153 v1.1-only rows read as политика, economy recall 0.730 to 0.448. All exact.

My bias fit lands close to yours: v1.3 0.640 and v1.1 0.673 over 25 seeds with both folds, a gap of 3.3 points instead of 2.7, probably down to how we fit. On emotions v1.3 stays ahead after the fit, as you found.

On the bootstrap: the raw gap is -8.4 points, 95% CI -11.2 to -5.5 by rows and -16.8 to -1.7 by labels. After the biases it is -3.2, CI -5.9 to -0.4 by rows and -6.3 to +0.3 by labels. So most of the drop is the label prior you describe. What remains does not clear zero at the label level, though with six labels that interval is coarse.

Yes, the biases belong beside T. They are fitted on the same labelled half and cost nothing at inference. I will add them to the calibration script.