Serkan Peker PRO

Neither, in these runs.

Gemma 4 12B, English output going from 256 to 349 tokens (two repetitions each): mean TPOT +1.8% (27.38 → 27.86 ms), median +1.2%, p99 unchanged (29.39 ms in both). The vLLM benchmark saved mean, median and p99, not p90.

Wall time does not stretch either. Slot occupancy (sum of request latencies ÷ (duration × 256)) is 0.992 and 0.995, and the run's duration grows ×1.357 against ×1.362 for the mean request.

Every request has exactly N output tokens, so the mean request takes TTFT + (N − 1) × mean TPOT, and throughput follows from that and occupancy. This reproduces the measured throughput ratio to three decimals for all four models. Per output token the time is TTFT/N + TPOT × (N − 1)/N, and only the first term shrinks with longer outputs. Its size is what separates the models: at 256 tokens TTFT is 15.7% of a Ministral request (919 ms against a 19.3 ms TPOT), 11.0% for Qwen3.8, 8.9% for Turkish-Gemma and 7.4% for Gemma 4 (559 ms against 27.4 ms).

Change in time per output token, 256 tokens → scaled length, from the TTFT term and the TPOT term:

• Ministral 3 14B: −5.1% and −0.8%
• Qwen3.8-27B: −3.2% and −3.0%
• Turkish-Gemma-9b-T1: −3.3% and +4.3%
• Gemma 4 12B: −1.8% and +1.7%

So the 5.6-point gap to Ministral splits into 3.2 points of prefill share, which Gemma 4 has less of to dilute, and 2.6 points of mean TPOT; occupancy moves it back by 0.2.

Why Gemma 4's TPOT rises less than Turkish-Gemma's at about the same KV per token (+0.48 ms against +0.95 ms mean TPOT) I have not isolated. One difference in the server logs: Gemma 4 ran on vLLM's TRITON_FLASH_ATTN backend with FlashAttention 4, Turkish-Gemma and Ministral on FLASH_ATTN with FlashAttention 3.

The runs kept per-request end-to-end latencies but not per-request TTFT and TPOT, so the TTFT share above is a mean over 4,096 requests per run, not a distribution. The script is projects/tr-serve/padded_tails.py in the GitHub repository; it reads the same raw files.

Thanks for the question. No translate-in, translate-out pipeline was measured here, so there is no quality number to add.

On cost, the serving data suggests the workaround rarely pays. At these loads the tax sits mostly on output tokens: the same answer takes 1.36 to 1.48 times as many tokens in Turkish on the four table models. A pipeline that ends by translating back into Turkish still generates those Turkish tokens, and it adds a pass over the Turkish input and another over the English draft.

It can come out cheaper in two cases. One is a translation step on a much smaller model than the one doing the task. The other is when most generated tokens never reach the user, as with long reasoning traces that could stay in English while only the final answer is in Turkish. Both are measurable, and the second matters more for current reasoning models. Neither is in this benchmark.

Your split checks out against the raw runs, and a follow-up run shows where the leftover comes from.

On each model's server: four runs at 256 concurrent requests, two repetitions each. Turkish with scaled output; English prompts padded to the token count of the matching Turkish prompt, with the same scaled output; unpadded English with the scaled output; English at 256 tokens. 2,245 of the 2,248 padded prompts match the Turkish token count exactly, the other three are one token longer.

Padded English runs as fast as Turkish. English ÷ Turkish throughput is 0.992 for Turkish-Gemma-9b-T1, 1.011 for Gemma 4 12B, 1.004 for Ministral 3 14B and 0.998 for Qwen3.8-27B. At equal sequence lengths the language itself costs nothing measurable here; the leftover is sequence length. In the same session it came out at 1.077, 1.061, 1.017 and 0.986.

The other two runs split it:

• Longer prompts at the same scaled output cost all four models a similar 5–8% (1.076, 1.054, 1.077, 1.052).
• Longer outputs with the same unpadded prompts give Ministral and Qwen3.8 about 6% back (0.940, 0.939), because the prefill is spread over more tokens. The two Gemma models get almost nothing back (1.009, 0.996).

That difference lines up with how much KV cache each decoded token reads (BF16 K and V): about 336 KiB for Turkish-Gemma (42 layers × 8 KV heads × 256), about 330 KiB for Gemma 4 (40 sliding layers × 8 × 256, plus 8 global layers with one 512-wide K/V head), 160 KiB for Ministral and 64 KiB for Qwen3.8 (16 full-attention layers; its 48 Gated DeltaNet layers keep a fixed-size state). When English output goes from 256 tokens to the scaled length, median TPOT rises 4.5% and 1.2% on the Gemma models and falls 0.7% and 3.1% on Ministral and Qwen3.8. The sliding windows (4,096 and 1,024 tokens) are longer than any Gemma sequence in these runs, all under 800 tokens, so they cap nothing.

The vLLM backend does not look like the common factor: Turkish-Gemma and Ministral both ran on FLASH_ATTN with FlashAttention 3 and sit at opposite ends.

Repeat runs agree within 2.4% in every cell. Raw runs and scripts are in the GitHub repository (projects/tr-serve/results/dolgu, padded_v3.sh, make_padded_prompts.py, padded_summary.py).

Thank you for the careful reading and the concrete suggestion. The run is done, and the result goes the other way.

Setup: the same four models, prompts and servers (one H200 each, vLLM 0.33.0, prefix caching off). English output stayed at 256 tokens. Turkish output was set to 256 × each model's token ratio (349 to 380 tokens), so both languages generate the same amount of content. As a control, Turkish at 256 tokens ran again on the same servers. At 256 concurrent requests there are four repetitions over two sessions, the second in reversed order; at 64 there are two.

TR/EN cost for the same content, Turkish output at 256 tokens → scaled:

256 concurrent requests

• Turkish-Gemma-9b-T1: 1.529 → 1.520
• Gemma 4 12B: 1.447 → 1.429
• Ministral 3 14B (BF16): 1.626 → 1.483
• Qwen3.8-27B: 1.517 → 1.406

64 concurrent requests

• Turkish-Gemma-9b-T1: 1.467 → 1.448
• Gemma 4 12B: 1.417 → 1.398
• Ministral 3 14B (BF16): 1.576 → 1.513
• Qwen3.8-27B: 1.482 → 1.414

The 256-token control reproduced the published figures within 0.5%. With scaled Turkish output, EN/TR per-token throughput at 256 requests drops from 1.06–1.10 to 0.99–1.09: the longer Turkish prompt is spread over more output tokens. For Gemma 4, Ministral and Qwen3.8 the scaled cost was lower in every repetition; for Turkish-Gemma the change stayed within run-to-run noise. Ministral was the noisiest model (up to 6.7% between runs), yet its highest scaled value, 1.575, is below all four of its 256-token values (1.603 to 1.732).

The KV cache was not a constraint at these loads; peak usage was 22% to 72%. Your point about KV growth should apply beyond that limit. Qwen3.8-27B on one GPU fills its cache at 512 concurrent requests, and longer Turkish sequences would get there sooner. These runs do not cover that regime.

The raw runs and the script will be part of the next dataset and GitHub update. Thanks again for raising this.