Noah P PRO

Yes, the next set will be stratified: every persona gets the same injection set (playful ones included), so a tie can't come from where the injections happened to land. We'll also split the metric into "refused but echoed the trick" vs "actually complied", with a hand-checked sample to validate the split, and fix the test (Fisher exact on real compliance) before running it. Target is 150 injection cases, which by your numbers is enough to separate ~6% from 0 if the base rate holds. Thanks for the careful read.

Good catch, and you're right that it's a 44-case subset. The newline bug hit the 4B baseline too, so I reran everything under the fixed template:
• Flagged injection success: base 6/44 (13.6%), tuned 1.7B 6/44 (13.6%), tuned 4B 3/44 (6.8%). So the "6x worse" direction doesn't hold; it's a tie.
• On clustering: the flag counts any echo of the injected words. All 6 tuned-1.7B hits are in-character refusals that name the trick ("I know no system prompt"), one per persona. The base model's hits include actually playing along ("Beep.", riffing on "cats"), mostly in playful personas.
• By hand, real compliance is 2–3/44 for the base and 0/44 for the tuned model.
With 44 cases the CIs are about ±15 points. The next eval will have 100+ injection cases and a metric that separates echo from compliance

Correction: two numbers above were wrong. Our first scoring run had a prompt-format bug (one newline instead of two after the empty think block the model was trained with). Re-scored with the right format:

• Prompt injections that worked: 2.3% → 14% (worse, not better; the tuned model sometimes echoes the injected words)
• Invented facts on trap questions: 20% → 10% (better, but not 3%)

Everything else holds after the fix: opener repetition 64% → ~19%, exact quotes first try 32% → ~100%, intent recovered ~80%, ~0.13 s per line. Injection resistance is the next thing we're training for.

If your interested in getting access to the model, join the alpha. https://p1ayer-1-gamedecider-playground.hf.space/alpha

Haha. If your interested in using it on a game of your own, join the alpha. https://p1ayer-1-gamedecider-playground.hf.space/alpha

Fair points, thanks. Honest denominators: the day-1 numbers were one card per type (intent 12/12, apology 7/14), because everyone gets the same five cards a day. As of now it’s 33 players and 10 cards: intent 18/18 over 2 cards, apology 9/19 over 2. So treat them as anecdotes, not results.

And yes, the reference label is the secret brief the writer was assigned. It measures recovering a scripted intent, not detecting real sincerity. The first-day cards were model-written to a brief and kept only if a second model read them blind the same way. Player-written lines started today.

Repeat guesses already only count once per player per card. I like the “can’t tell” idea, especially for apologies, where several intents fit the same words. I’ll add it, and publish the labeling rules plus per-card counts on the game page once there are enough cards to say anything.

1/4 is respectable, most people's first lie gets caught.

The trick is it isn't an LLM grading on a rubric. The judge is a small (~400M) encoder model trained only to answer game questions like "is this a threat / a lie against these facts / a sincere apology", and each challenge uses a small adapter trained just for that one question, so it's hard to talk it out of its job. It answers with a probability and can say "not sure" instead of guessing.

If you want the other direction, I made a second game where you are the judge: Mean It? — every day you read 5 lines from strangers and call whether they're faking it (apologies, threats, sarcasm, alibis), then you write one to fool tomorrow's players. Real players are currently 50/50 on the fake apologies:
https://p1ayer-1-gamedecider-playground.hf.space/mean?ref=hf-comment